[IPython-User] Parallel question: Sending data directly between engines

[Fernando Perez]

Hi Matthew,

On Fri, Jan 6, 2012 at 1:59 PM, Matthew Rocklin <mrocklin@gmail.com> wrote:
> Hello,
>
> What is the easiest way for me to send data directly between two ipython
> engines?
>
> I.e. if I'm running a "master" script on a particular machine and type in
> something like the following
>
> rc = p.Client()
> e0 = rc[0]
> e1 = rc[1]
> e0['data'] = e1['data']
>
> Then I suspect the data will be sent from e1 up to the master engine and
> then down to e0. How can I skip the intermediate step?

Yup, that's what happens now.  I know Min a little while ago whipped
up something for engine-to-engine communication in zmq, but there's no
high-level interface for that yet.  This implements a simple 2d
parallel wave solver, and it's the only example we have at this point
going in this direction:

https://github.com/ipython/ipython/tree/master/docs/examples/parallel/wave2D

Before we had punted on this problem altogether, b/c the overhead of
Twisted was so high that for all sensible use cases needing this, the
answer was always 'use mpi'.  Now, with zmq that's not the case
anymore, so we'll probably add some support for mpi-like
communications using purely zmq and the ipython api, but that hasn't
happened yet.


Best

f