[IPython-User] SSH remote ipcontroller?

MinRK benjaminrk@gmail....
Mon Apr 29 18:37:48 CDT 2013


> - Should the ipcontroller be run locally on my laptop or on a node of the
cluster?

Either way - it doesn't much matter

> - Is it possible to set up a remote cluster so it can be driven from an
ipython notebook on my laptop

Yes.

It does work, and those statements were both true, but the latter
disclaimer is out of date.  It used to be that the launchers didn't do any
file movement, so it would only work if

a) all the machines are on a shared filesystem
or
b) you move the files *once*, and then reuse them with `--reuse`

But more recently, the SSH launchers actually will push connection files
around, so the disclaimer about shared filesystems is out of date.

With the ssh launchers in particular, it can take a lot of non-default
config for it to work.  I have only really had good experiences with
ipcluster in local or batch (PBS, SGE, MPI, etc.) modes, as the SSH setup
is finicky, though it has always worked in my personal tests.



On Mon, Apr 29, 2013 at 2:25 PM, Jacob Vanderplas <jakevdp@cs.washington.edu
> wrote:

> Hi,
> I'm trying to set up my first ipython parallel configuration on a local
> compute cluster.  I've been following the ssh guidelines here:
>
> http://ipython.org/ipython-doc/stable/parallel/parallel_process.html#using-ipcluster-in-ssh-mode
> and have not had luck in getting it to work. I'm running into several
> problems which may or may not be related, but I wanted to ask a couple
> quick questions to make sure I'm on the right track:
>
> - Should the ipcontroller be run locally on my laptop or on a node of the
> cluster?
>
> - Is it possible to set up a remote cluster so it can be driven from an
> ipython notebook on my laptop? I noticed these two things in the docs:
>
> -------------------------
>
> The *ipcluster* command provides a simple way of starting a controller
> and engines in the following situations:
>
> 4. When the controller is started on localhost and the engines are started
> on remote nodes using *ssh*.
> ...
>
> Currently *ipcluster* requires that the IPYTHONDIR/profile_<name>/security directory
> live on a shared filesystem that is seen by both the controller and
> engines. If you don’t have a shared file system you will need to use *
> ipcontroller* and*ipengine* directly.
>
> -------------------------
>
> I'm confused by these two statements.  Can I use ipcluster to start a
> remote cluster, or not?  I've seen a couple demos where people drive remote
> clusters using the notebook interface on their laptop... does this require
> manually starting the controller and engines?
>
> Thanks for any guidance,
>    Jake
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130429/32dc2266/attachment-0001.html 


More information about the IPython-User mailing list