[IPython-User] Parallel question: Sending data directly between engines
Thu Jan 19 18:50:18 CST 2012
simple binary-tree engine interconnect example:
On Mon, Jan 16, 2012 at 22:34, MinRK <firstname.lastname@example.org> wrote:
> I've written a minimal binary-tree connector that provides all-reduce
> using PUSH/PULL sockets for the tree and PUB/SUB for the broadcast.
> It's quite simple, and almost all the logic is in establishing the
> connections in the first place, and is very similar to the existing
> EngineConnector example, just with different socket types and
> connection pattern (tree instead of all:all).
> I'll post it after I've cleaned it up a bit.
> I still think there's a more 'ØMQ-style' way to do this sort of thing,
> especially when ordering need not be enforced.
> On Sun, Jan 8, 2012 at 12:39, Brian Granger <email@example.com> wrote:
>> On Sun, Jan 8, 2012 at 12:30 PM, Olivier Grisel
>> <firstname.lastname@example.org> wrote:
>>> 2012/1/8 Brian Granger <email@example.com>:
>>>> Don't forget, you can always just use MPI/mpi4py with IPython, which
>>>> has very efficient reduce/allreduce that use the spanning tree
>>> Indeed but the MPI abstraction might be too strong (by hiding too much
>>> of the underlying computational runtime): in particular it might
>>> prevent the algorithm implementer to leverage data-locality by
>>> scheduling some tasks to be run where it knows the input data is
>>> already located (on the hard-drive or in shared memory as memory
>>> mapped file for instance) rather that shipping it over the network
>>> over and over again.
>>> I like the ability of the IPython engines and controller and client to
>>> collect whatever metadata they want, pass them around using pyzmq and
>>> make it possible to plug your own smart scheduler based on those
>>> runtime metadata.
>> Yes, IPython/pyzmq is definitely more flexible than MPI.
>>> I have not checked mpi4py recently though. Last time I experimented
>>> with mpi was more than 7 years ago so things might have changed.
>> It is truly an amazing package. Some of the best code I know of.
>>> http://twitter.com/ogrisel - http://github.com/ogrisel
>> Brian E. Granger
>> Cal Poly State University, San Luis Obispo
>> firstname.lastname@example.org and email@example.com
>> IPython-User mailing list
More information about the IPython-User