[IPython-user] Iterator version of TaskClient.map() that returns results as they become available?

[Jon Olav Vik]

I have lots of trivially parallel computations and will store the results to 
disk. Output order does not matter.

The TaskClient.map() function or TaskClient.parallel() decorator is convenient 
and load-balanced, but blocks until all the results are in. What I would like 
instead is a map()-like iterator that returns each result as it becomes 
available; similarly, an iparallel() decorator that returns an iterator. Then I 
could do:

from IPython.kernel import client
tc = client.TaskClient()
# Tasks that take an varying amount of time
import numpy as np
import time
@tc.iparallel()   # <-- nifty feature to be written
def f(x):
    time.sleep(np.random.random())
    return -x
N = 1000
for result in f(range(N)):
    print result # or save to file, or plot a data point

By flushing the output from time to time, I could then watch results take shape 
as they get computed.

I tried digging into the source code for TaskClient.map(), but was overwhelmed 
by the layers and intricacies of Twisted. Any pointers in the right direction 
would be highly appreciated.

Regards,
Jon Olav