[IPython-User] Parallel question: Sending data directly between engines

Matthew Rocklin mrocklin@gmail....
Fri Jan 6 22:05:24 CST 2012


Looks like there is another example in that same directory that creates
some convenience functions around inter-engine communication. I suspect it
will solve my problem.

https://github.com/ipython/ipython/blob/master/docs/examples/parallel/interengine/interengine.py



On Fri, Jan 6, 2012 at 5:30 PM, Fernando Perez <fperez.net@gmail.com> wrote:

> Hi Matthew,
>
> On Fri, Jan 6, 2012 at 1:59 PM, Matthew Rocklin <mrocklin@gmail.com>
> wrote:
> > Hello,
> >
> > What is the easiest way for me to send data directly between two ipython
> > engines?
> >
> > I.e. if I'm running a "master" script on a particular machine and type in
> > something like the following
> >
> > rc = p.Client()
> > e0 = rc[0]
> > e1 = rc[1]
> > e0['data'] = e1['data']
> >
> > Then I suspect the data will be sent from e1 up to the master engine and
> > then down to e0. How can I skip the intermediate step?
>
> Yup, that's what happens now.  I know Min a little while ago whipped
> up something for engine-to-engine communication in zmq, but there's no
> high-level interface for that yet.  This implements a simple 2d
> parallel wave solver, and it's the only example we have at this point
> going in this direction:
>
>
> https://github.com/ipython/ipython/tree/master/docs/examples/parallel/wave2D
>
> Before we had punted on this problem altogether, b/c the overhead of
> Twisted was so high that for all sensible use cases needing this, the
> answer was always 'use mpi'.  Now, with zmq that's not the case
> anymore, so we'll probably add some support for mpi-like
> communications using purely zmq and the ipython api, but that hasn't
> happened yet.
>
>
> Best
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120106/c1b53678/attachment.html 


More information about the IPython-User mailing list