[IPython-user] Iterator version of TaskClient.map() that returns results as they become available?
Thu Feb 11 14:25:55 CST 2010
> The TaskClient.map() function or TaskClient.parallel() decorator is
> and load-balanced, but blocks until all the results are in. What I would
> instead is a map()-like iterator that returns each result as it becomes
> available; similarly, an iparallel() decorator that returns an iterator.
> Then I
> could do:
> from IPython.kernel import client
> tc = client.TaskClient()
> # Tasks that take an varying amount of time
> import numpy as np
> import time
> @tc.iparallel() # <-- nifty feature to be written
> def f(x):
> return -x
> N = 1000
> for result in f(range(N)):
> print result # or save to file, or plot a data point
Currently this is not possible, but might be in the future. We could
modify barrier in a slight manner to make it possible. Could you file
a feature request here:
> By flushing the output from time to time, I could then watch results take
> as they get computed.
No doubt, this would be very nice.
> I tried digging into the source code for TaskClient.map(), but was
> by the layers and intricacies of Twisted. Any pointers in the right
> would be highly appreciated.
Yes, the code is pretty complicated right now and definitely uses
The place to start looking is in:
In particular look at the code for barrier. I would have to think
about it more, but we really want a barrier method that has a mode
that will return a list of task ids that are complete at the time
barrier is called, rather than waiting for all tasks to be finished.
Might not be too difficult to implement this.
> Jon Olav
> IPython-user mailing list
More information about the IPython-user