[IPython-User] Parallel Setup: Can connect engines over network, but not run Hello World

MinRK benjaminrk@gmail....
Thu Oct 6 17:50:12 CDT 2011


On Wed, Oct 5, 2011 at 23:31, Markus Baden <markus.baden@gmail.com> wrote:

> Hi list,
>
> I'm trying to use IPython together with our High Performance Computing
> facilties at NUS. The jobs are distributed via LSF, so I intend to follow
> the discussion on setting up IPython with LSF [1]. Before using the LSF
> setup I wanted to test things by just having remote engines on their
> computer, controlled by a local controller on my laptop.
>
> However, I have the problem that there is no shared network drive between
> the computers that are used for submitting the jobs (which would run the
> engines) and my laptop. So far I was able to get a local ipcontroller
> running and remote ipengines that show up as connected. However
> c[:].apply_snyc(lambda: 'Hello World') did not work.
>
> Here is what I did:
>
> - ipython profile create --parallel  --profile=hpc_test
> - set c.HubFactory.ip = '*' in ipcontroller_config.py
> - run on laptop ipcontroller --profile=hpc_test&
> - create hpc_test profile on remote computer
> - transfer ipcontroller-engine.json from laptop to remote computer
> - run ipengine --profile=hpc_test on remote computer, which led the
> ipcontroller on my laptop to acknowledge that an engine was connected
> - on local machine start ipython, c = Client(profile='hpc_test')
> - c.ids shows the correct ids, but c[:].apply_snyc(lambda: 'Hello World')
> just hangs
>
> Since I am totally new to this, I'm super confused and don't know where to
> proceed from here. Somehow I was expecting that I have to tell the engine
> over which protocol to talk to the controller. I was intending to use ssh
> for that, but somehow I could connect engines without even specifying this.
>

If you specified 'ip=*', that means you instructed the controller to listen
on all interfaces. If the machines are visible to each other, then there is
no need to use SSH tunnels.  The ip=* is not always reliable with zeromq, so
sometimes using 0.0.0.0 instead (which means the same thing) works better.

It's also possible that specifying the external IP itself will be necessary,
depending on your network.

When you use the 0.0 or * IP, then IPython uses a 'location' value, to
determine what IP to use, and whether the controller is local or not.  You
can see this value in the ipcontroller-engine.json, and you can set it with
the `--location` argument, if IPython's guess is wrong.

If you want to be using SSH tunnels, then you typically specify this at the
controller, with '--enginessh=[user@]<server>', but it is only necessary if
ports are not visible from one machine to another.

The main things to look at are the listening ports (the 'Hub * addrs'
controller log output, and the 'location' value in ipcontroller-engine.json.
 When debugging, adding the '--debug' flag to ipcontroller and ipengine may
help enlighten things.

-MinRK


>
> Could somebody point me in the right direction, on where to proceed from
> here on? Any hints are greatly appreciated.
>
> Thanks a lot,
>
> Markus
>
> [1] http://thread.gmane.org/gmane.comp.python.ipython.user/6487
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20111006/f322d2b0/attachment.html 


More information about the IPython-User mailing list