[IPython-user] Iterator version of TaskClient.map() that returns results as they become available?

Brian Granger ellisonbg.net@gmail....
Thu Feb 11 14:25:55 CST 2010


> The TaskClient.map() function or TaskClient.parallel() decorator is
> convenient
> and load-balanced, but blocks until all the results are in. What I would
> like
> instead is a map()-like iterator that returns each result as it becomes
> available; similarly, an iparallel() decorator that returns an iterator.
> Then I
> could do:

> from IPython.kernel import client
> tc = client.TaskClient()
> # Tasks that take an varying amount of time
> import numpy as np
> import time
> @tc.iparallel()   # <-- nifty feature to be written
> def f(x):
>     time.sleep(np.random.random())
>     return -x
> N = 1000
> for result in f(range(N)):
>     print result # or save to file, or plot a data point

Currently this is not possible, but might be in the future.  We could
modify barrier in a slight manner to make it possible.  Could you file
a feature request here:


> By flushing the output from time to time, I could then watch results take
> shape
> as they get computed.

No doubt, this would be very nice.

> I tried digging into the source code for TaskClient.map(), but was
> overwhelmed
> by the layers and intricacies of Twisted. Any pointers in the right
> direction
> would be highly appreciated.

Yes, the code is pretty complicated right now and definitely uses
Twisted heavily.

The place to start looking is in:


In particular look at the code for barrier.  I would have to think
about it more, but we really want a barrier method that has a mode
that will return a list of task ids that are complete at the time
barrier is called, rather than waiting for all tasks to be finished.
Might not be too difficult to implement this.



> Regards,
> Jon Olav
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user

More information about the IPython-user mailing list