[IPython-User] Installing IPython on a computing cluster?

MinRK benjaminrk@gmail....
Tue Mar 19 23:20:13 CDT 2013


On Tue, Mar 19, 2013 at 2:53 PM, Skipper Seabold <jsseabold@gmail.com>wrote:

> Hi,
>
> I'm trying to convince our HPC admin to install IPython.parallel, and I'm
> not at all confident that I've grasped all the details. I'm hoping someone
> might be able to help me out here and either correct me or fill in as
> needed.
>
> My understanding is that we need to have IPython installed on the log-in
> node (the host), but it also needs to be installed on the worker nodes (the
> engines). Following the usual instructions for each machine.
>
>
> http://ipython.org/ipython-doc/dev/install/install.html#dependencies-for-ipython-parallel-parallel-computing
>
> The only extra dependency we will need is (py)zmq.
>
> Optionally, we will need to install mpi4py to support MPI applications.
>

>
> http://ipython.org/ipython-doc/dev/parallel/parallel_mpi.html#additional-installation-requirements
>

That's exactly right.


>
> After this is done, we will have to follow the instructions here to create
> an appropriate profile on the log-in node in the user's config directory
> and then copy the json file over the config directories on the other nodes.
> The users won't have write access to the nodes, so I'm a bit confused about
> best practice here or maybe I'm misunderstanding.
>

Hm, that's a new restriction I haven't dealt with.  Is there no shared
filesystem, etc.? Can the nodes mount remote filesystems?
If not, you may need to write a new engine startup script that loads
connection info from some other mechanism, rather than the filesystem.
Should be doable, but a bit weird.


>
>
> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#starting-the-controller-and-engines-on-different-hosts
>
> Then I should just be able to use the LSFLauncher to schedule jobs?
>
>
> http://ipython.org/ipython-doc/dev/api/generated/IPython.parallel.apps.launcher.html#IPython.parallel.apps.launcher.LSFLauncher
>

Yes, if you can get connection files to a location readable by the nodes,
then LSF launcher should work.


>
> Thanks,
>
> Skipper
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130319/83c2f1bd/attachment.html 


More information about the IPython-User mailing list