[IPython-user] RPC and MultiEngineClient

Matthew Turk matthewturk@gmail....
Tue Mar 23 12:48:45 CDT 2010


Hi there,

I'm trying to figure out if something is possible inside IPython, and
thought this might be the best place to ask.

Right now, we've been approaching parallel computation using the
MultiEngineClient interface in IPython on top of mpi4py.  This has
worked very well - the integration with the batch system, the
abstraction of essentially all of the security and authentication
processes, and the ease of distributing tasks and operations to the
remote processes has been very nice.  What we'd like to do is to
extend our current interface -- using "%px" and "%autopx" to instead
utilize proxy objects directly.

Right now we set up identical objects on every engine, which then
handle the domain decomposition and IO through MPI-based decomposition
-- this way we end up using the same interface both in interactive
parallel (where the controller is IPython) and scripted parallel
(where the controller is a single script, executed roughly in lockstep
on all the processes).

What would simplify this process greatly would be the ability to
register proxy objects on both ends -- I'm thinking mainly of the
XML-RPC model, where arguments are passed over the wire, rather than
strings evaluated remotely.

Is it possible, even using Foolscap or the XML-RPC stdlib module, to
use the same infrastructure that the MultiEngineClient sets up to set
up an RPC channel?  Any suggestions on where to start digging?

Thanks!

-Matt


More information about the IPython-user mailing list