[IPython-User] Parallel question: Sending data directly between engines

[Fernando Perez]

Hi Olivier,

On Fri, Jan 6, 2012 at 9:58 PM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
> Very interesting thread, thanks for sharing. Let me share some of my
> related use cases with to feed the IPython.parallel community with
> some additional use cases you might not be aware of.

[...]

Thanks a lot for this explanation and the link.  I think I must have
missed something, because it seems to me that we already offer that:

In [26]: from IPython.parallel import Client
    ...: v = Client()[:]

In [27]: x = rand(len(v.targets))
    ...: v.scatter('x', x)
Out[27]: <AsyncResult: scatter>

In [29]: v['x']
Out[29]:
[array([ 0.55751144]),
 array([ 0.51093798]),
 array([ 0.03574132]),
 array([ 0.31362035])]

In [30]: v['x_sum'] = sum(v['x'])
    ...: v['x_sum']
Out[30]:
[1.4178110846551577,
 1.4178110846551577,
 1.4178110846551577,
 1.4178110846551577]


No?  That last line performs what the all_reduce operation is defined
as: it pulls 'x' from each node, does the reduction, and sends back
the global reduction to all nodes now named x_sum.

Can you clarify what I'm not understanding?

Thanks!

f