[IPython-User] ipython parallelism and closures

MinRK benjaminrk@gmail....
Mon Apr 23 15:15:04 CDT 2012


On Mon, Apr 23, 2012 at 03:24, Andrew Jaffe <a.h.jaffe@gmail.com> wrote:

> Dear all,
>
> I have what seems to be a very simply parallelizable task:
>
>     ghdirs = [ './y/_x_%d/' % isim for isim in range(*nsim) ]
>     gh = map(lambda ghd:
>                     GHcalc.getGH(freq=freq, num=num, nc=nc, GHdir=ghd,
>                        verbose=False, no_offset_correction=True),
>              ghdirs)
>
> where the variables freq, num and nc are defined prior to this block.
>
> Because of the presence of these variables, this is a closure, which is
> unpicklable, and therefore cannot be sent to ipython clients, hence I
> get the message "ValueError: Sorry, cannot pickle code objects with
> closures" when I replace map with dview.map_sync.
>
> I am sure there must be a trivial way to parallize this... Can anyone help?
>

You just have to pass those values as arguments to your map function.  They
can even be passed as default kwargs in your lambda:

ghdirs = [ './y/_x_%d/' % isim for isim in range(*nsim) ]
    gh = map(lambda ghd, freq=freq, num=num, nc=nc:
                    GHcalc.getGH(freq=freq, num=num, nc=nc, GHdir=ghd,
                       verbose=False, no_offset_correction=True),
             ghdirs)

-MinRK


>
> Andrew
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120423/e68b2069/attachment.html 


More information about the IPython-User mailing list