[IPython-user] RPC and MultiEngineClient

Brian Granger ellisonbg@gmail....
Wed Mar 24 12:08:34 CDT 2010


Great question.  It is still a bit awkward, but it is possible to
build proxy objects.

Here is what I have been doing.

1. Create a local object for the proxy:

class FooProxy(object):

  def __init__(self, mec):
    self.mec = mec  # this is a live multiengine client
    self.execute('remote_foo = Foo()')

2. Then when you want to proxy a method, just use mec.push/execute/pull.

  def bar(self, x):
    mec.execute('result = remote_foo.bar(x)')
    result = mec.pull('x')
    return result

Really, this pattern of push, execute, pull needs to be encapsulated
into a single step as it becomes really common when you are building
proxies.  But this should at least get you started.  Now you just have
to decide what engines the remote object is on, etc.  Let me know if
you have questions.



On Tue, Mar 23, 2010 at 10:48 AM, Matthew Turk <matthewturk@gmail.com> wrote:
> Hi there,
> I'm trying to figure out if something is possible inside IPython, and
> thought this might be the best place to ask.
> Right now, we've been approaching parallel computation using the
> MultiEngineClient interface in IPython on top of mpi4py.  This has
> worked very well - the integration with the batch system, the
> abstraction of essentially all of the security and authentication
> processes, and the ease of distributing tasks and operations to the
> remote processes has been very nice.  What we'd like to do is to
> extend our current interface -- using "%px" and "%autopx" to instead
> utilize proxy objects directly.
> Right now we set up identical objects on every engine, which then
> handle the domain decomposition and IO through MPI-based decomposition
> -- this way we end up using the same interface both in interactive
> parallel (where the controller is IPython) and scripted parallel
> (where the controller is a single script, executed roughly in lockstep
> on all the processes).
> What would simplify this process greatly would be the ability to
> register proxy objects on both ends -- I'm thinking mainly of the
> XML-RPC model, where arguments are passed over the wire, rather than
> strings evaluated remotely.
> Is it possible, even using Foolscap or the XML-RPC stdlib module, to
> use the same infrastructure that the MultiEngineClient sets up to set
> up an RPC channel?  Any suggestions on where to start digging?
> Thanks!
> -Matt
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo

More information about the IPython-user mailing list