[IPython-User] ipython parallelism and closures

[Andrew Jaffe]

Hi Min,

On 23/04/2012 21:15, MinRK wrote:
>
>
> On Mon, Apr 23, 2012 at 03:24, Andrew Jaffe <a.h.jaffe@gmail.com
> <mailto:a.h.jaffe@gmail.com>> wrote:
>
>     Dear all,
>
>     I have what seems to be a very simply parallelizable task:
>
>          ghdirs = [ './y/_x_%d/' % isim for isim in range(*nsim) ]
>          gh = map(lambda ghd:
>                          GHcalc.getGH(freq=freq, num=num, nc=nc, GHdir=ghd,
>                             verbose=False, no_offset_correction=True),
>                   ghdirs)
>
>     where the variables freq, num and nc are defined prior to this block.
>
>     Because of the presence of these variables, this is a closure, which is
>     unpicklable, and therefore cannot be sent to ipython clients, hence I
>     get the message "ValueError: Sorry, cannot pickle code objects with
>     closures" when I replace map with dview.map_sync.
>
>     I am sure there must be a trivial way to parallize this... Can
>     anyone help?
>
> You just have to pass those values as arguments to your map function.
>   They can even be passed as default kwargs in your lambda:
>
> ghdirs = [ './y/_x_%d/' % isim for isim in range(*nsim) ]
>      gh = map(lambda ghd, freq=freq, num=num, nc=nc:
>                      GHcalc.getGH(freq=freq, num=num, nc=nc, GHdir=ghd,
>                         verbose=False, no_offset_correction=True),
>               ghdirs)
>
> -MinRK


Thanks! This is obvious in retrospect -- though I'm embarrassed that 
after something like 7 years of python programming, I don't think I've 
ever used (and may not have even known about) keyword arguments in lambdas.

I would think that this is such a common use case that it might be worth 
documenting either in the 'caveats: closures' section, or in the 
discussion of dview.map. (And it gets a bit ugly when you have a block 
to parallelize with a lot of "free variables" of this type)

Andrew