[IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook

Gael Varoquaux gael.varoquaux@normalesup....
Sun Feb 8 04:22:00 CST 2009


On Sat, Feb 07, 2009 at 10:09:05AM -0800, Brian Granger wrote:
> > I am not using any thread in the wx GUI. This was a very important design
> > choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
> > possible the terminal reactive when a computation is going on, but there
> > are no threads (except when a subprocess is used, but I was very careful
> > to make them play well with the mainloop).

> Because we are likely moving in this direction, I want to understand
> this better.  What do you mean by "fiddling with the mainloop?"  Is
> this "fiddling" optional?  If it causes bugs in your GUI (is this what
> you are saying?) why do you do it?  You imply that the terminal was
> not responsive until you did this extra stuff.  Can you say a bit
> more?

Sorry, I was out.

When I say fiddling with the event loop, I mean many things. The big deal
is that you cannot get a refresh, or a key event without running the
event loop. The user is executing code, and executing it in the event
loop, bocking it, as we are going non-multithreaded, but it is
imperative, sequential code, and we really need event-driven code that
yields to the event-loop in order to get refreshes and events. So I
overloaded a few operations, such as print and raw_input to yield back to
the event loop. Now I couldn't simply use wx.Yield blindly, because it is
fragile, and getting this right was a bit tricky. 

For screen refresh, you want to refresh ASAP, because you don't want
debug statements of a long running calculation to pop up only after it
finish. So I had to trigger controlled refresh events to do refreshes on
prints. See method 'write', line 135 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/console_widget.py
This method is injected in a sys.stdout and sys.stderr replacement. I had
to be careful not to get a refresh called when we where already in a
wx.Yield. This is why I had to add the refresh keyword, and use it
cleverly. In addition, printing a lot of text to the terminal would slow
down a lot, because a refresh is expensive. This is why I introduced the
notion of a delayed refresh: there is a timer, and a queue. Trying to
refresh triggers the timer, and the refresh happens delayed, so that the
queue fills in. Obviously this means that if you are unlucky and the
write method does not get called for a long time while you where waiting
for the timer to empty, you don't get a refresh, because you don't enter
the event loop again.

I had to be careful with all other possible interactions, such as
keyboard callbacks, not to trigger the Yield. Also, I had to write an
OS-level output redirection that would weave the captured flux for
compiled code with the Python flux, and call the refresh callback ASAP,
this is done in
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/kernel/core/redirector_output_trap.py

For the raw_input, I had to start a second event loop, to process
keyboard event while the main event loop was blocked by the user code
executing. See method 'raw_input', line 176 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
There is readline-like logic that is used here to determine when this
second event loop is terminated and the user code can resume.

Finally, the trickiest part was running subprocesses. The reason being
that I needed both synchronous prints, and key-event processing. The
subprocess in run in a separate thread, see
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/_process/pipedprocess.py
Its input stream is fed in by a raw_input-like mechanism, with a separate
event-loop running, see method system_call, line 197 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
. I have to use a separate event-loop because the call to a subprocess is
blocking for the user code. The output stream is polled (yeah, I am not
proud of the polling, I couldn't figure out something better), and fed to
a 'buffered_write' method. No need to do a wx.Yield, here, as an
event-loop is running in a separate thread. Which just need to reduce the
number of write called per unit of time to avoid flooding the event loop,
and to make sure we are thread-safe, see method 'buffered_write', line
264 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
The termination of the subprocess is wired to a callback which flushes
the buffer and terminates the inner event loop.

I believe that this is mostly it. There is a lot more of little subtle
things you have to be careful with: not calling your callbacks in
infinite loops, making sure methods that can be called in other thread
are thread safe, making sure you don't screw the order of events. This is
quite tricky code, I believe. The good news is that all event loops are
quite similar (the Tk one is different, in Python, as it runs in a
separate OS thread, but we can forget that, and use it like the others).
It would be possible to build an object that would abstract scintilla and
the event-loop control out of different GUI toolkits, and expose a common
interface. This would actually be fairly easy and would get us an
interactive frontend for all toolkits.

Gaël


More information about the IPython-dev mailing list