[IPython-User] parallel load-balanced question

Toby Burnett tburnett@uw....
Fri Oct 21 19:57:46 CDT 2011

Thanks for the response! That works nicely, but it seems to wait for each task to finish in the order started, and does not seem provide access to the metadata (I want to be able to examine the stdout for each process as it finishes.) Could I set up a callback for finishing tasks with the result and metadata?


From: MinRK [mailto:benjaminrk@gmail.com]
Sent: Friday, October 21, 2011 15:18
To: Toby Burnett
Cc: ipython-user@scipy.org
Subject: Re: [IPython-User] parallel load-balanced question

On Fri, Oct 21, 2011 at 14:44, Toby Burnett <tburnett@uw.edu<mailto:tburnett@uw.edu>> wrote:
Hi, I’m converting to the new parallel scheme, very nice.
I start a number of tasks with the lview.map, and would like to be able to extract some results from finished tasks before they all finish. The AsyncMapResult object does not seem to allow that, however. It is a list with an entry for each task, but no way that I could find to get the metadata for a finished task before all had finished. Am I missing something? Doing this the wrong way?

AsyncMapResult is *iterable* before everything is done, but you can't do index-access until they are all done:

amr = view.map(myfunc, myargs)

for result in amr:
    dostuff(result) # this is the actual return value of myfunc(myargs[i])

Or, for a full toy example, load-balanced, super-slow RMS with intermediate status updates:

from random import random
from IPython import parallel

rc = parallel.Client()
v = rc.load_balanced_view()

def slow_square(x):
    import time
    return x*x

amr = v.map_async(slow_square, randoms)

for n,r in enumerate(amr):
    sum_sq += r
    print "intermediate rms %i: %.3f" % (n, sqrt(sum_sq / (n+1)))

Since the AsyncMapResult object is just an iterable of its results, you can just use it with builtin functions like sum(amr), reduce(func, amr), map(process_result, amr) etc.  These functions will advance step by step as results arrive.


--Toby Burnett

IPython-User mailing list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20111022/52678224/attachment.html 

More information about the IPython-User mailing list