[IPython-User] parallel with ssh

Toby Burnett tburnett@uw....
Wed Oct 26 07:20:00 CDT 2011


A follow-up on a previous thread, and a difference with version 0.10.

First the previous “ipcluster ssh –clusterfile …” worked nicely with only the need to specify a simple file.  Now it is necessary to specify not only the set of engines, but to set both engine and controller options for communication: perhaps the code that implements ipcluster could either set this, or clearly warn the user?

But my advice, thanks MinRK,  is to avoid ipcluster and start the controller and engines myself, in fact necessary if the controller and engines are not on a shared filesystem.  I’ve done so in a class with a loop like this

        for host, n in self.engines.items():
            for i in range(n):
                cmd = 'ssh %s@%s ipengine --file=%s&'%(self.user,host,self.json_file)
   os.system(cmd))
                time.sleep(delay)

It works, but leaves a lot of background ssh processes on the machine where this runs: studying the very elegant code in IPython/parallel/apps/launcher.py I’m sure that it should be easy to start n engines on host m by creating an SSHEngineSetLauncher. Am I right?

--Toby
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20111026/d09c88e4/attachment.html 


More information about the IPython-User mailing list