[IPython-user] Any obvious way to have an ipcontroller listening on two devices?

Brian Granger ellisonbg.net@gmail....
Mon Sep 22 23:17:27 CDT 2008


Brant,

> I've been using ipython 0.9b3 and the ipcontroller/ipengine
> parallelization scheme with amazon EC2, and pretty much everything
> about the experience so far has been fantastic, but I have one "dream"
> feature that I haven't figured out how to get to work.

Your dreams have come true (I think it should work)...see below.

> The layout is as follows:
> amazon encourages you to use an "internal" network device and ip
> wherever possible for communication between the instances (nodes), and
> especially with mpi, the internal interface provides a fairly
> substantial speedup to 'talky' parallel code.  This is no problem, and
> one can use amazon's internal interface hostnames in the
> cluster-config file used with ipcluster -f to boot an ipcluster
> controller/engine complement on EC2 with no trouble.
> However, I would also like to be able to use my work computer as the
> client, which is to say connecting to the controller via either the
> multiengineclient or taskclient from outside EC2, and therefore via
> the public interface, with the public hostname for the machine on
> which ipcontroller is running.

There are command line flags to the ipcontroller command that control
the ip addresses (interfaces) listed in the furl files.  Here is an
example:

bgranger$ ipcontroller --engine-location=192.168.0.1
--client-location=192.168.0.2

bgranger$ head *.furl
==> ipcontroller-engine.furl <==
pb://okgja7ykg55eeirdw6hwrxwqbonjqtvs@192.168.0.1:64033/r5ikpw55564cpirjtxk753o2ume7cfrt

==> ipcontroller-mec.furl <==
pb://mfmkfzyxkoc33n7aeaekub3ey3f2gdid@192.168.0.2:64032/2euq7zyllfezv6bgo2xwcz6cnr5tdb4g

==> ipcontroller-tc.furl <==
pb://mfmkfzyxkoc33n7aeaekub3ey3f2gdid@192.168.0.2:64032/cqvm2wgp44cqjg4j73526wqneypmohcy

Here I am just pretending that the private interface is on 192.168.0.1
and the public one is 0.2, but I think you will get the idea.  Also,
by default, the controller will listen on all interfaces, so it should
be listening on both the public and private.  You simply need to give
the hints about what furl corresponds to which interface.  Please try
this and let us know if you have problems.

A few important points though:

* We have released a stable version of IPython 0.9.1 that has a number
of bug fixes relative to the beta you are using...

* But, the release does not have the ipcluster -f option for starting
the cluster (which you are using).  We were having problems with this,
so we disabled it until we can do a proper implementation.  Obviously,
if it is working for you, feel free to keep using it.  The ipcluster
for the beta should work fine with the 0.9.1 release.  We are in the
middle of reimplementing the stuff in ipcluster from the ground up in
a way that will make starting things _much_ easier.  Watch the list
for details.

Cheers,

Brian

> The problem this presents seems to be that you only get one valid set
> of .furl files, and these point to one controller IP/hostname (i.e.
> for the engines to register with the controller).  How do I get a set
> of  .furls such that I can have engines registering with the private
> IP/hostname, and still connect to the controller via the public
> interface?  Is this possible?
>
> Thank you for your time!
> -Brant
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-user mailing list