[IPython-User] stream data from engine

Brian Burns brian.p.burns@gmail....
Fri Feb 15 13:24:10 CST 2013


Hello,

I'm wondering if this I'm approaching this problem correctly or not.  I've
got an ipython engine running on a remote machine.  I would like to stream
data from that engine to the client and for the client to be able turn that
stream off at a later point.

My scenario is I'd like to have an engine return results from a tailable
mongodb cursor.   Here is one approach I've taken.

1. get a directview instance to remote engine.
2. load module on engine which has functions for doing mongo tail
3. tail function looks something like this utilizing publish_data in kernel
zmq package.

def tail(query={}):
    c = mongo_collection.find(query, await_data=True, tailable=True)
    while c.isalive:
        publish_data(c.next())


4. from the client I can call this function an get an asyncresult back.
5. I can continually call data on the result and everything works fine.

The problem comes when I want to turn the tail off.  I can't invoke any
more function calls because the tail function is essentially running
forever and blocking.  So I naturally tried executing a portion of the code
in a thread.  The problem with that approach is the tail function returns
and publish_data calls from the running thread are no longer bound to the
asyncresult.

Once the tail function returns the thread is still executing on the engine,
where does the data from publish_data actually go?  If I can't get to it
from asyncresult how else would I get to it?  Would i have to dig into the
guts of the zmq IOPub channel ?

Regards,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130215/5ed9ed12/attachment.html 


More information about the IPython-User mailing list