[IPython-User] Scattering functions using IPython

reckoner reckoner@gmail....
Wed Aug 8 18:05:08 CDT 2012


>  Message: 4
>  Date: Wed, 8 Aug 2012 12:10:33 -0700
>  From: MinRK <benjaminrk@gmail.com>
>  Subject: Re: [IPython-User] Scattering functions using IPython
>     parallel
>
>  > Hi,
>  >
>  > I'm trying to use IPython's scatter mechanism to distribute a large
>  > number of dynamically generated functions, but I'm stymied by pickling
>  > errors. Some furious googling about this issue reveals that pickling
>  > functions is problematic, but is there anything in the IPython kit that
>  > can make this work? It would be a lot of work to restructure my entire
>  > codebase that relies heavily on these dynamically generated functions.
>  > For reference, the functions themselves do not rely on global variables
>  > in the namespace and are otherwise pretty self-contained.
>  >
>
>  Are you using master?  If so, try applying PR
>  2246<https://github.com/ipython/ipython/pull/2246>,
>  and see if it helps. The serialization is being redone a bit, and it is
>  more conservative about inspecting arguments.  2246 fixes an issue where it
>  was a little *too* conservative.
>
>  We can handle simple functions, but most importantly not ones with
>  closures.  What error do you get?

Here's an example:


    from IPython import parallel
    rc = parallel.Client()
    dv=rc[:]
    flist=[ eval('lambda x:sin(2*x-%3.3f)'% d) for d in linspace(0,1,10)]
    dv.scatter('foo',flist)
    PicklingError: Can't pickle <type 'function'>: attribute lookup 
__builtin__.function failed


Thanks!





More information about the IPython-User mailing list