[IPython-dev] Twisted reactor + IPython0

Brian Granger ellisonbg.net@gmail....
Thu Mar 20 17:15:14 CDT 2008


>  >  It would certainly be great to have this done in ip0 as an optional
>  >  component: we could always tell our users: you have two options for
>  >  gui support:
>
>  It's in trunk now. It's still a threads-based hack, in that ipython
>  mainloop runs in a thread, but now the mainloop is a twisted reactor.
>  See IPython/twshell.py. You can try it out by "python twshell.py".
>  kbdint and Exit seem to work.
>
>  I used the stuff from mtexp branch, so macros should work, and it
>  should survive reactor death (5 sec timeout as before). Now, I gotta
>  run, but someone might want to take a look at it.

I should provide a few more details about my earlier objections to
having this in ipython0.

One of the big problems we have dealt with in IPython1 is that we
want/need to have networking clients that are based on Twisted.  Thus,
we need to have the Twisted reactor running in the IPython0 frontend.
But that is the easy part.  The challenge is that we need to put a
blocking interface on top of Twisted deferred returning functions and
methods.  This took us a _very_ long time to figure out how to do.
Here is what we are doing currently in ipython1 (when using it from
ipython0):

1) We simply start the twisted reactor in a thread.

2) When we need to call a method/function that returns a deferred, we
use blockingCallFromThread.  This uses a Queue to block the users
thread (where Ipython0 is running) until the deferred has fired in the
other thread (the twisted one).

There code for these things is here:

http://bazaar.launchpad.net/~ipython/ipython/ipython1-dev/annotate/ellisonbg%40gmail.com-20080311024040-ctltp13xxrh9iba7?file_id=twistedutil.py-20080302203151-17poomviavsf2zjj-141

Or if you have the ipython1-dev branch, it is in:

ipython/kernel/twistedutil.py

>From our testing so far, this seems to provide a fairly robust way of
running the reactor in ipython0.

...queue scary music...

But, I am not sure how all of this works with the current threaded
Ipython shells.

I think most of it works, but it probably fails to handle signals
properly.  Also, I am not sure how everything would behave if we run a
Twisted GUI reactor.

Here is another big problem lurking on the horizon.

Eventually, your average IPython session will be an IPython1 session
and will thus have the Twisted reactor running in its main thread.  We
want to be able to write networking clients that will run in that
context.  That itself is not a problem.  The problem is that we want
to be able to have blocking network clients.  This means somehow we
need to figure out a way of blocking on the result of a deferred.

With a single thread, this is impossible.  The only way I know of
doing this is to have two threads:

1) One running the twisted reactor

2) Another that runs the users code

Then user code can use blockingCallFromThread to block on a deferred.
While we don't yet have this implemented in Ipython1, it should not be
difficult.

The bigger complication comes in when you add a GUI event loop to the
picture.  Then you need to make sure that all three things (GUI event
loop, twisted reactor and user code that potentially needs to block on
a deferred) work together in a thread happy manner.

That is the main thing to keep in mind.  Sorry about my earlier rant.
I don't want to discourage people from playing around with this stuff.
 But, this stuff is extremely complicated and easy to break.  Please
just keep in mind that currently we are in this really odd state where
IPython1 needs to be able to work from within IPython0.

Cheers,

Brian


>
>  --
>  Ville M. Vainio - vivainio.googlepages.com
>  blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
>  _______________________________________________
>
>
> IPython-dev mailing list
>  IPython-dev@scipy.org
>  http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>


More information about the IPython-dev mailing list