[IPython-User] parallelism with arbitrary function handles

MinRK benjaminrk@gmail....
Tue Jun 19 21:54:21 CDT 2012


On Tue, Jun 19, 2012 at 7:14 PM, Robert Nishihara <robertnishihara@gmail.com
> wrote:

> Hi,
>
> I have been running into a problem using map_sync in IPython.parallel
> (please tell me if this is the wrong place to ask this question). The
> script below works fine:
>
>         from IPython import parallel
>
>         def run_stuff():
>             def func(x):
>                 return x
>             rc = parallel.Client(profile='sge')
>             dview = rc[:]
>             dview['func'] = func
>             print dview.map_sync(func, range(10))
>
>         run_stuff()
>
> The issue occurs when I want "func" to be parametrized by some arbitrary
> function "f". The script below is the simple example that illustrates the
> failure I am seeing
>
>         from IPython import parallel
>
>         def run_stuff(f):
>             rc = parallel.Client(profile='sge')
>             dview = rc[:]
>             dview['f'] = f
>             def func(x,f=f):
>                 return f(x)
>             print dview.map_sync(func, range(10))
>
>         def g(x):
>             return x
>
>          run_stuff(g)
>
> This script fails with the error:
>
>         AttributeError: 'module' object has no attribute 'g'.
>
> The name "g" is not accessible to the function "run_stuff". The two
> scripts are attached. Can anyone help?
>

Here's a detailed
explanation<http://stackoverflow.com/questions/10857250/python-name-space-issues-with-ipython-parallel>
of
how namespaces are resolved.

To breakdown your case:

def func(x, f=f)

creates a function with default value tied to the module locals (module
being __main__).  g is not defined remotely, so __main__.g doesn't resolve.
 We have some utilities that allow functions to be serialized in a way that
works around this, but func.func_defaults is  inspected and wrapped like
func itself.  If we added that, then this code would just work, I think.

If you avoid passing arguments via the func_defaults, and pass them as
arguments explicitly, you avoid the problem:

def func(x,f):
...

view.map_sync(func, range(10), [f]*10)

Will avoid the issue.

There is no need to push f before passing it to map.

-MinRK


>
> -Robert
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120619/e3ff988c/attachment.html 


More information about the IPython-User mailing list