[IPython-User] Scattering functions using IPython

[MinRK]

On Wed, Aug 8, 2012 at 4:05 PM, reckoner <reckoner@gmail.com> wrote:

>
> >  Message: 4
> >  Date: Wed, 8 Aug 2012 12:10:33 -0700
> >  From: MinRK <benjaminrk@gmail.com>
> >  Subject: Re: [IPython-User] Scattering functions using IPython
> >     parallel
> >
> >  > Hi,
> >  >
> >  > I'm trying to use IPython's scatter mechanism to distribute a large
> >  > number of dynamically generated functions, but I'm stymied by pickling
> >  > errors. Some furious googling about this issue reveals that pickling
> >  > functions is problematic, but is there anything in the IPython kit
> that
> >  > can make this work? It would be a lot of work to restructure my entire
> >  > codebase that relies heavily on these dynamically generated functions.
> >  > For reference, the functions themselves do not rely on global
> variables
> >  > in the namespace and are otherwise pretty self-contained.
> >  >
> >
> >  Are you using master?  If so, try applying PR
> >  2246<https://github.com/ipython/ipython/pull/2246>,
> >  and see if it helps. The serialization is being redone a bit, and it is
> >  more conservative about inspecting arguments.  2246 fixes an issue
> where it
> >  was a little *too* conservative.
> >
> >  We can handle simple functions, but most importantly not ones with
> >  closures.  What error do you get?
>
> Here's an example:
>
>
>     from IPython import parallel
>     rc = parallel.Client()
>     dv=rc[:]
>     flist=[ eval('lambda x:sin(2*x-%3.3f)'% d) for d in linspace(0,1,10)]
>     dv.scatter('foo',flist)
>     PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
>

I can confirm that the above-mentioned PR #2246 fixes this issue

You can even avoid the eval, with:

flist = [ lambda x,offset=d: sin(2*x - offset) for d in linspace(0,1,10) ]


It should also work just fine with 0.13, though I haven't checked.

-MinRK


>
> Thanks!
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120808/9ceed4ae/attachment.html