[IPython-User] Parallel question: Sending data directly between engines

[Fernando Perez]

On Sat, Jan 7, 2012 at 10:59 PM, MinRK <benjaminrk@gmail.com> wrote:
>
> You can even do full map/reduce in one line with view.map_async and the
> builtin reduce function.  The map will be evaluated remotely and in
> parallel, but the reduce will be evaluated locally, if asynchronously:
>
> def reduce_func(a,b):
>     """simplest reduce func: sum"""
>     return a+b
>
> def map_func(n):
>     """take about a millisecond to return a random integer"""
>     import random
>     time.sleep(.001*random.random())
>     return random.randint(n)
>
> reduce(reduce_func, view.map_async(map_func, range(10000)))
>
> is equivalent to:
>
> reduce(reduce_func, map(map_func, range(10000)))
>
> And for load-balanced views, this can even be done out of order in case
> early results may take more time to arrive than later ones:
>
> reduce(reduce_func, lbview.map_async(map_func, range(10000,1,-1),
> ordered=False))

Très, très cool...  Mandatory new example :)

f