[IPython-User] Installing IPython on a computing cluster?

Skipper Seabold jsseabold@gmail....
Wed Mar 20 12:38:12 CDT 2013


On Wed, Mar 20, 2013 at 12:20 AM, MinRK <benjaminrk@gmail.com> wrote:

>
>
> On Tue, Mar 19, 2013 at 2:53 PM, Skipper Seabold <jsseabold@gmail.com>wrote:
>
>> Hi,
>>
>> I'm trying to convince our HPC admin to install IPython.parallel, and I'm
>> not at all confident that I've grasped all the details. I'm hoping someone
>> might be able to help me out here and either correct me or fill in as
>> needed.
>>
>> My understanding is that we need to have IPython installed on the log-in
>> node (the host), but it also needs to be installed on the worker nodes (the
>> engines). Following the usual instructions for each machine.
>>
>>
>> http://ipython.org/ipython-doc/dev/install/install.html#dependencies-for-ipython-parallel-parallel-computing
>>
>> The only extra dependency we will need is (py)zmq.
>>
>> Optionally, we will need to install mpi4py to support MPI applications.
>>
>
>>
>> http://ipython.org/ipython-doc/dev/parallel/parallel_mpi.html#additional-installation-requirements
>>
>
> That's exactly right.
>

Hi Min,

Great. I'm not as confused as I thought then.


>
>
>>
>> After this is done, we will have to follow the instructions here to
>> create an appropriate profile on the log-in node in the user's config
>> directory and then copy the json file over the config directories on the
>> other nodes. The users won't have write access to the nodes, so I'm a bit
>> confused about best practice here or maybe I'm misunderstanding.
>>
>
> Hm, that's a new restriction I haven't dealt with.  Is there no shared
> filesystem, etc.? Can the nodes mount remote filesystems?
> If not, you may need to write a new engine startup script that loads
> connection info from some other mechanism, rather than the filesystem.
> Should be doable, but a bit weird.
>

There is access to a shared filesystem. I didn't think about this, but it
makes perfect sense now.

Thanks,

Skipper


>
>
>>
>>
>> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#starting-the-controller-and-engines-on-different-hosts
>>
>> Then I should just be able to use the LSFLauncher to schedule jobs?
>>
>>
>> http://ipython.org/ipython-doc/dev/api/generated/IPython.parallel.apps.launcher.html#IPython.parallel.apps.launcher.LSFLauncher
>>
>
> Yes, if you can get connection files to a location readable by the nodes,
> then LSF launcher should work.
>
>
>>
>> Thanks,
>>
>> Skipper
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130320/62dbd9d3/attachment.html 


More information about the IPython-User mailing list