[IPython-User] Installing IPython on a computing cluster?

MinRK benjaminrk@gmail....
Wed Mar 20 14:28:11 CDT 2013


On Wed, Mar 20, 2013 at 10:38 AM, Skipper Seabold <jsseabold@gmail.com>wrote:

> On Wed, Mar 20, 2013 at 12:20 AM, MinRK <benjaminrk@gmail.com> wrote:
>
>>
>>
>> On Tue, Mar 19, 2013 at 2:53 PM, Skipper Seabold <jsseabold@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to convince our HPC admin to install IPython.parallel, and
>>> I'm not at all confident that I've grasped all the details. I'm hoping
>>> someone might be able to help me out here and either correct me or fill in
>>> as needed.
>>>
>>> My understanding is that we need to have IPython installed on the log-in
>>> node (the host), but it also needs to be installed on the worker nodes (the
>>> engines). Following the usual instructions for each machine.
>>>
>>>
>>> http://ipython.org/ipython-doc/dev/install/install.html#dependencies-for-ipython-parallel-parallel-computing
>>>
>>> The only extra dependency we will need is (py)zmq.
>>>
>>> Optionally, we will need to install mpi4py to support MPI applications.
>>>
>>
>>>
>>> http://ipython.org/ipython-doc/dev/parallel/parallel_mpi.html#additional-installation-requirements
>>>
>>
>> That's exactly right.
>>
>
> Hi Min,
>
> Great. I'm not as confused as I thought then.
>
>
>>
>>
>>>
>>> After this is done, we will have to follow the instructions here to
>>> create an appropriate profile on the log-in node in the user's config
>>> directory and then copy the json file over the config directories on the
>>> other nodes. The users won't have write access to the nodes, so I'm a bit
>>> confused about best practice here or maybe I'm misunderstanding.
>>>
>>
>> Hm, that's a new restriction I haven't dealt with.  Is there no shared
>> filesystem, etc.? Can the nodes mount remote filesystems?
>> If not, you may need to write a new engine startup script that loads
>> connection info from some other mechanism, rather than the filesystem.
>> Should be doable, but a bit weird.
>>
>
> There is access to a shared filesystem. I didn't think about this, but it
> makes perfect sense now.
>


Then you should be set - just make sure the profile_dir you use is
accessible on the shared filesystem from both the controller and engines.

-MinRK


>
> Thanks,
>
> Skipper
>
>
>>
>>
>>>
>>>
>>> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#starting-the-controller-and-engines-on-different-hosts
>>>
>>> Then I should just be able to use the LSFLauncher to schedule jobs?
>>>
>>>
>>> http://ipython.org/ipython-doc/dev/api/generated/IPython.parallel.apps.launcher.html#IPython.parallel.apps.launcher.LSFLauncher
>>>
>>
>> Yes, if you can get connection files to a location readable by the nodes,
>> then LSF launcher should work.
>>
>>
>>>
>>> Thanks,
>>>
>>> Skipper
>>>
>>> _______________________________________________
>>> IPython-User mailing list
>>> IPython-User@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>
>>>
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130320/08035235/attachment.html 


More information about the IPython-User mailing list