[IPython-User] ipython parallel task depencencies

Peter Prettenhofer peter.prettenhofer@gmail....
Wed Sep 25 00:45:02 CDT 2013


I'd like to use ipython parallel for task parallelism with task
dependencies. The tasks look as follows:

1: n ``fetch`` tasks are submitted.
2: after a fetch task is complete I submit k ``extract-transform`` tasks
per fetch result.
3: Barrier that waits until all ``extract-transform`` tasks are complete
4: m ``load`` tasks are submitted that reduce the results of the
``extract-transform`` tasks.

I'm not quite sure how this can be represented in IPython parallel. In
particular, the dependency between 1 and 2 without a barrier between the
two. One possibility would be to use async results and add a callback to
the ``fetch`` task that submits the ``extract-transform`` tasks but that's
not possible since AsyncResult does not support callbacks yet.


Peter Prettenhofer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130925/86de1a3b/attachment.html 

More information about the IPython-User mailing list