[IPython-User] parallel load-balanced question

MinRK benjaminrk@gmail....
Fri Oct 21 17:18:25 CDT 2011


On Fri, Oct 21, 2011 at 14:44, Toby Burnett <tburnett@uw.edu> wrote:

>  Hi, I’m converting to the new parallel scheme, very nice.****
>
> I start a number of tasks with the lview.map, and would like to be able to
> extract some results from finished tasks before they all finish. The
> AsyncMapResult object does not seem to allow that, however. It is a list
> with an entry for each task, but no way that I could find to get the
> metadata for a finished task before all had finished. Am I missing
> something? Doing this the wrong way?
>

AsyncMapResult is *iterable* before everything is done, but you can't do
index-access until they are all done:

amr = view.map(myfunc, myargs)

for result in amr:
    dostuff(result) # this is the actual return value of myfunc(myargs[i])

Or, for a full toy example, load-balanced, super-slow RMS with intermediate
status updates:

from random import random
from IPython import parallel

rc = parallel.Client()
v = rc.load_balanced_view()

def slow_square(x):
    import time
    time.sleep(1)
    return x*x

amr = v.map_async(slow_square, randoms)

sum_sq=0
for n,r in enumerate(amr):
    sum_sq += r
    print "intermediate rms %i: %.3f" % (n, sqrt(sum_sq / (n+1)))

Since the AsyncMapResult object is just an iterable of its results, you can
just use it with builtin functions like sum(amr), reduce(func, amr),
map(process_result, amr) etc.  These functions will advance step by step as
results arrive.

-MinRK

 ****
>
> ** **
>
> --Toby Burnett****
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20111021/8e61dba7/attachment.html 


More information about the IPython-User mailing list