[IPython-User] parallel load-balanced question

MinRK benjaminrk@gmail....
Fri Oct 21 21:32:43 CDT 2011


On Fri, Oct 21, 2011 at 17:57, Toby Burnett <tburnett@uw.edu> wrote:

>  Thanks for the response! That works nicely, but it seems to wait for each
> task to finish in the order started, and does not seem provide access to the
> metadata (I want to be able to examine the stdout for each process as it
> finishes.) Could I set up a callback for finishing tasks with the result and
> metadata?
>

Yes, it does iterate in order.  I have unordered iteration working locally,
for which I will issue a Pull Request, probably this weekend.

Note that AsyncResults are merely convenient wrappers for results and
metadata, they are not the canonical store.  That is the Client itself,
which has dictionaries for each, keyed by msg_id.  This allows you to
actually watch stdout/stderr for a task without even waiting for that single
task to finish.  An important part of the parallel code is that the *only*
things you need to get everything about a job are its id and a Client.

If you want to do something with tasks that AsyncResults don't provide, you
can just get their msg_ids attribute (a list of message ID strings), and do
what you like with them.

If you would rather have a collection of AsyncResult objects rather than a
single AMR, you can do that, with:

asyncresults = [ rc.get_result(msg_id, block=False) for msg_id in
amr.msg_ids ]
# which is now a list of individual AsyncResult objects, one for each
*chunk* of the AsyncMapResult (no)

An important thing to note if you do split up an AsyncMapResult is that the
individual results will always return a list of length `chunksize`, even if
chunksize is 1.

A quick example <https://gist.github.com/1305415> of performing custom
handling of results as they arrive (regardless of submission order),
including printing stdout and some metadata.

-MinRK

****
>
> ** **
>
> --Toby****
>
> ** **
>
> ** **
>
> *From:* MinRK [mailto:benjaminrk@gmail.com]
> *Sent:* Friday, October 21, 2011 15:18
> *To:* Toby Burnett
> *Cc:* ipython-user@scipy.org
> *Subject:* Re: [IPython-User] parallel load-balanced question****
>
> ** **
>
> ** **
>
> On Fri, Oct 21, 2011 at 14:44, Toby Burnett <tburnett@uw.edu> wrote:****
>
> Hi, I’m converting to the new parallel scheme, very nice.****
>
> I start a number of tasks with the lview.map, and would like to be able to
> extract some results from finished tasks before they all finish. The
> AsyncMapResult object does not seem to allow that, however. It is a list
> with an entry for each task, but no way that I could find to get the
> metadata for a finished task before all had finished. Am I missing
> something? Doing this the wrong way?****
>
> ** **
>
> AsyncMapResult is *iterable* before everything is done, but you can't do
> index-access until they are all done:****
>
> ** **
>
> amr = view.map(myfunc, myargs)****
>
> ** **
>
> for result in amr:****
>
>     dostuff(result) # this is the actual return value of myfunc(myargs[i])
> ****
>
> ** **
>
> Or, for a full toy example, load-balanced, super-slow RMS with intermediate
> status updates:****
>
> ** **
>
> from random import random****
>
> from IPython import parallel****
>
> ** **
>
> rc = parallel.Client()****
>
> v = rc.load_balanced_view()****
>
> ** **
>
> def slow_square(x):****
>
>     import time****
>
>     time.sleep(1)****
>
>     return x*x****
>
> ** **
>
> amr = v.map_async(slow_square, randoms)****
>
> ** **
>
> sum_sq=0****
>
> for n,r in enumerate(amr):****
>
>     sum_sq += r****
>
>     print "intermediate rms %i: %.3f" % (n, sqrt(sum_sq / (n+1)))****
>
> ** **
>
> Since the AsyncMapResult object is just an iterable of its results, you can
> just use it with builtin functions like sum(amr), reduce(func, amr),
> map(process_result, amr) etc.  These functions will advance step by step as
> results arrive.****
>
> ** **
>
> -MinRK****
>
> ** **
>
>   ****
>
> --Toby Burnett****
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user****
>
>  ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20111021/0d2b9bf2/attachment-0001.html 


More information about the IPython-User mailing list