[IPython-user] Latency and the MultiEngineClient

Matthew Turk matthewturk@gmail....
Thu Apr 29 20:41:15 CDT 2010

Hi there,

Let me start off by saying the MultiEngineClient has been very useful
for me in my research, particularly via the ipcluster interface using
MPI as an interconnect.

What I'd like to do is slightly repurpose it to act as a fancy proxy
to a computation engine that, rather than being expensive in
computation, is expensive in data (on disk) and memory requirements --
so it makes sense to keep the data on the high-end machine and the
client interface on the local machine.  In fact, this is to support
building a local Chaco GUI that interfaces with a remote data store.

What I'm seeing, though, with my simple tests on a single machine
hosting the ipengine, the ipcontroller and the actual client script,
is that the latency for this sort of behavior:

from IPython.kernel import client
mec = client.get_multiengine_client()

mec.execute("some_var = result", block = False)
var = mec.pull("some_var")

ends up having higher latency than I expected, and by far most of the
time (in that script, not in either the ipcontroller, as I don't know
how to profile either the ipengine or the ipcontroller) is spend
waiting for the the thread to acquire in _blockFromThread in the
MultiEngineClient.  The remote routine takes roughly 0.05 seconds, but
the process of execute/pull takes roughly 1 second per operation.  So
the GUI is responsive, but sluggish.  Still, despite that, I think
it's pretty cool that IPython makes this so easy.

I guess the simple question really is, is it possible to get
low-latency behavior with this kind of setup?  Have I made a mistake
in the way I've set it up -- would a different setup work better?

Thanks very much,


More information about the IPython-user mailing list