[IPython-user] Iterator version of TaskClient.map() that returns results as they become available?

Brian Granger ellisonbg.net@gmail....
Thu Feb 11 14:25:55 CST 2010


Jon,

> The TaskClient.map() function or TaskClient.parallel() decorator is
> convenient
> and load-balanced, but blocks until all the results are in. What I would
> like
> instead is a map()-like iterator that returns each result as it becomes
> available; similarly, an iparallel() decorator that returns an iterator.
> Then I
> could do:

> from IPython.kernel import client
> tc = client.TaskClient()
> # Tasks that take an varying amount of time
> import numpy as np
> import time
> @tc.iparallel()   # <-- nifty feature to be written
> def f(x):
>     time.sleep(np.random.random())
>     return -x
> N = 1000
> for result in f(range(N)):
>     print result # or save to file, or plot a data point

Currently this is not possible, but might be in the future.  We could
modify barrier in a slight manner to make it possible.  Could you file
a feature request here:

https://bugs.launchpad.net/ipython

> By flushing the output from time to time, I could then watch results take
> shape
> as they get computed.

No doubt, this would be very nice.

> I tried digging into the source code for TaskClient.map(), but was
> overwhelmed
> by the layers and intricacies of Twisted. Any pointers in the right
> direction
> would be highly appreciated.

Yes, the code is pretty complicated right now and definitely uses
Twisted heavily.

The place to start looking is in:

IPython.kernel.task
IPython.kernel.taskclient
IPython.kernel.taskfc

In particular look at the code for barrier.  I would have to think
about it more, but we really want a barrier method that has a mode
that will return a list of task ids that are complete at the time
barrier is called, rather than waiting for all tasks to be finished.
Might not be too difficult to implement this.

Cheers,

Brian

> Regards,
> Jon Olav
>
>
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-user mailing list