[IPython-User] IPython cluster with PBS when only head node is accessible

Darlan Cavalcante Moreira darcamo@gmail....
Wed Oct 23 14:49:36 CDT 2013


I haven't tried using IPython parallel with PBS yet, but I'm very
interested in this. My use case is exactly like yours (only head node
available), but the PBS cluster does not have IPython or any of the
essential python libraries installed, such as numpy (something I still
have to find a way around, since I'm the only one using python for work
- others use MATLAB, which is installed in all nodes).


As I understand, with the approach you described you can't shutdown your
workstation, since the controller would stop. Furthermore, if there are
only a few free cores then PBS will start some of the submitted engines,
while others will be queued. Is this right? I imagine this approach
would be good for iterative work on your workstation, but I wonder
what's the best approach for long running jobs.


In fact, I have many questions after reading the documentation in
http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode
For instance, where the pbs profile should be created? In the cluster's
head node or in my machine? I imagine the profile should be created in
the head node and then I would need to run the ipcluster command in the
head node through SSH from my machine (probably using something like
screen so that the ipcluster command is not stopped when I close the SSH
connection from my machine).


-- 
Darlan Cavalcante Moreira
darcamo@gmail.com


mluessi@gmail.com writes:

> ..OK, I managed to get it working. The solution is actually quite
> simple. What I did is start the controller manually on my workstation,
> i.e.,
>
> ipcontroller --profile=pbs
>
> and on the head node of the cluster I run a bash script with a
> for-loop that uses "qsub" to start a script with "ipengine
> --profile=pbs" for each engine I want to start. The engines are then
> started on the nodes and they connect to the controller running on my
> workstation :).
>
> On Tue, Oct 22, 2013 at 10:51 AM, Martin Luessi <mluessi@gmail.com> wrote:
>> Hi,
>>
>> I have been using IPython cluster in SSH mode for a while and it works
>> perfectly. It allows me to distribute jobs to a bunch of workstations
>> that would otherwise sit idle during the night :).
>>
>> Anyways, I now have a larger job and I would like to run it on a
>> "proper cluster". We use a PBS system and I have configured an IPython
>> profile as described in
>>
>> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode
>>
>> I can successfully launch the cluster and engines from the head node
>> of the cluster. However, the problem is that the nodes of the cluster
>> are only accessible through the head node, i.e., I can't make a direct
>> SSH connection from my workstation (where IPython notebook is running)
>> to one of the nodes. Is there a way to still use the IPython cluster
>> in this case without making changes to the network setup? This issue
>> has been raised on stackoverflow:
>>
>> http://stackoverflow.com/questions/5186989/parallel-ipython-ipcluster-through-head-node
>>
>> but it seems there was no solution, so I thought I would ask here.
>>
>> Thanks,
>>
>> Martin
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user



More information about the IPython-User mailing list