[IPython-User] Parallel question: Sending data directly between engines

Brian Granger ellisonbg@gmail....
Sun Jan 8 14:01:10 CST 2012


Don't forget, you can always just use MPI/mpi4py with IPython, which
has very efficient reduce/allreduce that use the spanning tree
approach.

On Sun, Jan 8, 2012 at 11:02 AM, Fernando Perez <fperez.net@gmail.com> wrote:
> Hi Olivier,
>
> On Sun, Jan 8, 2012 at 4:26 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
>> AFAIK the traditional way to implement the AllReduce is the to first a
>> spanning tree over the nodes / engines. For instance if you have 10
>> nodes, define a fixed arbitrary binary tree that spans all the nodes
>> involved in the computation:
>>
>> 0
>
> Thanks for the explanation.  Indeed, we don't have communication
> patterns with other topologies implemented yet out of the box, so it's
> good to have this use case described well for us.  It seems like we
> should be able to use the star-topology for now, which works out of
> the box (but has the limitation you point out), and implementing other
> communication patterns such as a spanning tree one should be easy.
>
> Do you know how they handle node failure?  What happens when a node
> disappears to its children?
>
> Cheers,
>
> f
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com


More information about the IPython-User mailing list