[IPython-User] IPython cluster with PBS when only head node is accessible

Martin Luessi mluessi@gmail....
Tue Oct 22 16:49:31 CDT 2013


..OK, I managed to get it working. The solution is actually quite
simple. What I did is start the controller manually on my workstation,
i.e.,

ipcontroller --profile=pbs

and on the head node of the cluster I run a bash script with a
for-loop that uses "qsub" to start a script with "ipengine
--profile=pbs" for each engine I want to start. The engines are then
started on the nodes and they connect to the controller running on my
workstation :).

On Tue, Oct 22, 2013 at 10:51 AM, Martin Luessi <mluessi@gmail.com> wrote:
> Hi,
>
> I have been using IPython cluster in SSH mode for a while and it works
> perfectly. It allows me to distribute jobs to a bunch of workstations
> that would otherwise sit idle during the night :).
>
> Anyways, I now have a larger job and I would like to run it on a
> "proper cluster". We use a PBS system and I have configured an IPython
> profile as described in
>
> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode
>
> I can successfully launch the cluster and engines from the head node
> of the cluster. However, the problem is that the nodes of the cluster
> are only accessible through the head node, i.e., I can't make a direct
> SSH connection from my workstation (where IPython notebook is running)
> to one of the nodes. Is there a way to still use the IPython cluster
> in this case without making changes to the network setup? This issue
> has been raised on stackoverflow:
>
> http://stackoverflow.com/questions/5186989/parallel-ipython-ipcluster-through-head-node
>
> but it seems there was no solution, so I thought I would ask here.
>
> Thanks,
>
> Martin


More information about the IPython-User mailing list