[IPython-User] parallelism with arbitrary function handles

MinRK benjaminrk@gmail....
Tue Jun 19 22:06:34 CDT 2012


Also, your code works exactly as-is with this
change<https://github.com/ipython/ipython/pull/1990> (the
`view['f'] = f` step remains unnecessary).

-MinRK

On Tue, Jun 19, 2012 at 7:54 PM, MinRK <benjaminrk@gmail.com> wrote:

> On Tue, Jun 19, 2012 at 7:14 PM, Robert Nishihara <
> robertnishihara@gmail.com> wrote:
>
>> Hi,
>>
>> I have been running into a problem using map_sync in IPython.parallel
>> (please tell me if this is the wrong place to ask this question). The
>> script below works fine:
>>
>>         from IPython import parallel
>>
>>         def run_stuff():
>>             def func(x):
>>                 return x
>>             rc = parallel.Client(profile='sge')
>>             dview = rc[:]
>>             dview['func'] = func
>>             print dview.map_sync(func, range(10))
>>
>>         run_stuff()
>>
>> The issue occurs when I want "func" to be parametrized by some arbitrary
>> function "f". The script below is the simple example that illustrates the
>> failure I am seeing
>>
>>         from IPython import parallel
>>
>>         def run_stuff(f):
>>             rc = parallel.Client(profile='sge')
>>             dview = rc[:]
>>             dview['f'] = f
>>             def func(x,f=f):
>>                 return f(x)
>>             print dview.map_sync(func, range(10))
>>
>>         def g(x):
>>             return x
>>
>>          run_stuff(g)
>>
>> This script fails with the error:
>>
>>         AttributeError: 'module' object has no attribute 'g'.
>>
>> The name "g" is not accessible to the function "run_stuff". The two
>> scripts are attached. Can anyone help?
>>
>
> Here's a detailed explanation<http://stackoverflow.com/questions/10857250/python-name-space-issues-with-ipython-parallel> of
> how namespaces are resolved.
>
> To breakdown your case:
>
> def func(x, f=f)
>
> creates a function with default value tied to the module locals (module
> being __main__).  g is not defined remotely, so __main__.g doesn't resolve.
>  We have some utilities that allow functions to be serialized in a way that
> works around this, but func.func_defaults is  inspected and wrapped like
> func itself.  If we added that, then this code would just work, I think.
>
> If you avoid passing arguments via the func_defaults, and pass them as
> arguments explicitly, you avoid the problem:
>
> def func(x,f):
> ...
>
> view.map_sync(func, range(10), [f]*10)
>
> Will avoid the issue.
>
> There is no need to push f before passing it to map.
>
> -MinRK
>
>
>>
>> -Robert
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120619/edf71049/attachment-0001.html 


More information about the IPython-User mailing list