[IPython-User] Parallel question: Sending data directly between engines
Tue Jan 17 00:34:42 CST 2012
I've written a minimal binary-tree connector that provides all-reduce
using PUSH/PULL sockets for the tree and PUB/SUB for the broadcast.
It's quite simple, and almost all the logic is in establishing the
connections in the first place, and is very similar to the existing
EngineConnector example, just with different socket types and
connection pattern (tree instead of all:all).
I'll post it after I've cleaned it up a bit.
I still think there's a more 'ØMQ-style' way to do this sort of thing,
especially when ordering need not be enforced.
On Sun, Jan 8, 2012 at 12:39, Brian Granger <firstname.lastname@example.org> wrote:
> On Sun, Jan 8, 2012 at 12:30 PM, Olivier Grisel
> <email@example.com> wrote:
>> 2012/1/8 Brian Granger <firstname.lastname@example.org>:
>>> Don't forget, you can always just use MPI/mpi4py with IPython, which
>>> has very efficient reduce/allreduce that use the spanning tree
>> Indeed but the MPI abstraction might be too strong (by hiding too much
>> of the underlying computational runtime): in particular it might
>> prevent the algorithm implementer to leverage data-locality by
>> scheduling some tasks to be run where it knows the input data is
>> already located (on the hard-drive or in shared memory as memory
>> mapped file for instance) rather that shipping it over the network
>> over and over again.
>> I like the ability of the IPython engines and controller and client to
>> collect whatever metadata they want, pass them around using pyzmq and
>> make it possible to plug your own smart scheduler based on those
>> runtime metadata.
> Yes, IPython/pyzmq is definitely more flexible than MPI.
>> I have not checked mpi4py recently though. Last time I experimented
>> with mpi was more than 7 years ago so things might have changed.
> It is truly an amazing package. Some of the best code I know of.
>> http://twitter.com/ogrisel - http://github.com/ogrisel
> Brian E. Granger
> Cal Poly State University, San Luis Obispo
> email@example.com and firstname.lastname@example.org
> IPython-User mailing list
More information about the IPython-User