[IPython-User] Asynchronous Plotting

[Michael McNeil Forbes]

This question has been asked several times, but I can find no  
satisfactory answer:

How can I have plots generated in the background while a calculation  
runs in the main thread (so that I can interrupt the calculation with  
Ctrl-C for example.)  My goals are:

1) Main calculation does not block when calling a "plot" function to  
plot data as it is computed.
2) Main calculation can be interrupted by user with Ctrl-C.
3) Plotting occurs in "background" plotting.
4) Cross-platform.  (At least on Mac OS X and Linux).
5) Simple.

I envision a multithreaded approach (see below), but 2) seems to  
require that the calculation be in the main thread, which the plotting  
GUI's do not like.  A multiprocessing approach seems like it should  
also work (like http://stackoverflow.com/a/4662511/1088938), but on my  
Mac (10.5) I get

The process has forked and you cannot use this CoreFoundation  
functionality safely. You MUST exec().
Break on  
__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ 
() to debug.

Also, I think that the GIL is probably good here so I do not have to  
copy data between processes, nor do I need to worry about the  
calculation thread updating the data while the plot thread is reading  
etc.

I feel there must be some sort of simple solution with IPython, but I  
will be darned if I can figure it out.  It seems like the pre 1.11  
ipython solution of running the calculation in a secondary thread (by  
calling embed()) but I cannot figure out how to get user interrupts to  
affect that thread (point 2).

Would something like twisted help?

I know I could run two separate python instances, and use sockets to  
communicate back and forth, but this approach seems much more  
complicated (the user needs to start both processes for example).   
Maybe I could use parallelpython to automate this, but it still seems  
significantly more complicated than it should be.

Any suggestions would be greatly appreciated.

Here is what I envision would work if I could run the plotting in a  
secondary thread: The computational thread calls update() which pushes  
the arguments into a queue discarding unused previous values.  The  
plotting thread should get these from the queue and plot as quickly as  
it can.

In fact, this seems to work on linux with ipython --pylab wx, but I  
can't figure out how to get this working on my Mac (the plot window  
never update).  It also fails with other backends (probably because I  
am trying to plot from a non-main thread.)

Thanks,
Michael.

import time
from threading import Thread, Event
from Queue import Queue, Empty, Full
import warnings

#from multiprocessing import Process as Thread
#from multiprocessing import Queue, Event

from matplotlib import pyplot as plt


class PylabPlot(Thread):
    def __init__(self, timeout=60):
        r"""
        Parameters
        ----------
        timeout : float
           Kill the thread if no activity for this many seconds.
        """
        self.timeout = timeout
        self.queue = Queue(maxsize=1)
        self.event = Event()
        Thread.__init__(self)
        self.dead = False
        self.running = False
        self.start()

    def stop(self):
        self.running = False
        self.event.set()

    def update(self, *v, **kw):
        if self.dead:
            warnings.warn("Plot thread is dead... ignoring update  
command.")
            return
        try:
            self.queue.put((v, kw), block=False)
        except Full:
            # Remove an item, then put this one.
            try:     # draw might be called in the meantine...
                self.queue.get(block=False)
            except Empty:
                pass
            self.queue.put((v, kw), block=False)  # This should not fail
        self.event.set()

    def run(self):
        self.running = True
        while self.running:
            self.event.wait(timeout=self.timeout)
            self.event.clear()
            if self.queue.empty():
                self.running = False
            else:
                while not self.queue.empty():
                    try:
                        v, kw = self.queue.get(block=False)
                    except Empty:
                        pass
                    self.draw(*v, **kw)
        self.dead = True


def go():
    import numpy as np

    class Plot(PylabPlot):
        def draw(self, x, y):
            plt.ioff()
            plt.clf()
            plt.plot(x, y)
            print y
            plt.ion()
            plt.draw()
            time.sleep(0.5)     # Pretend this is slow

    plt.clf()
    tmp = raw_input("Position plot window, and press enter to run...")
    x = np.linspace(0, 2*np.pi, 10)
    dt = 0.001
    t = 0
    p = Plot()
    try:
        for n in xrange(10000):
            t += dt
            y = np.sin(x - t)
            p.update(x=x, y=y)   # This should not take long and  
should not block
    finally:
        p.stop()   # Make sure that we exit gracefully.

if __name__ == "__main__":
    go()