[IPython-User] parallelism with arbitrary function handles

[Robert Nishihara]

Ah, this fixes everything. Thank you!

On Tue, Jun 19, 2012 at 11:06 PM, MinRK <benjaminrk@gmail.com> wrote:

> Also, your code works exactly as-is with this change<https://github.com/ipython/ipython/pull/1990> (the
> `view['f'] = f` step remains unnecessary).
>
> -MinRK
>
>
> On Tue, Jun 19, 2012 at 7:54 PM, MinRK <benjaminrk@gmail.com> wrote:
>
>> On Tue, Jun 19, 2012 at 7:14 PM, Robert Nishihara <
>> robertnishihara@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been running into a problem using map_sync in IPython.parallel
>>> (please tell me if this is the wrong place to ask this question). The
>>> script below works fine:
>>>
>>>         from IPython import parallel
>>>
>>>         def run_stuff():
>>>             def func(x):
>>>                 return x
>>>             rc = parallel.Client(profile='sge')
>>>             dview = rc[:]
>>>             dview['func'] = func
>>>             print dview.map_sync(func, range(10))
>>>
>>>         run_stuff()
>>>
>>> The issue occurs when I want "func" to be parametrized by some arbitrary
>>> function "f". The script below is the simple example that illustrates the
>>> failure I am seeing
>>>
>>>         from IPython import parallel
>>>
>>>         def run_stuff(f):
>>>             rc = parallel.Client(profile='sge')
>>>             dview = rc[:]
>>>             dview['f'] = f
>>>             def func(x,f=f):
>>>                 return f(x)
>>>             print dview.map_sync(func, range(10))
>>>
>>>         def g(x):
>>>             return x
>>>
>>>          run_stuff(g)
>>>
>>> This script fails with the error:
>>>
>>>         AttributeError: 'module' object has no attribute 'g'.
>>>
>>> The name "g" is not accessible to the function "run_stuff". The two
>>> scripts are attached. Can anyone help?
>>>
>>
>> Here's a detailed explanation<http://stackoverflow.com/questions/10857250/python-name-space-issues-with-ipython-parallel> of
>> how namespaces are resolved.
>>
>> To breakdown your case:
>>
>> def func(x, f=f)
>>
>> creates a function with default value tied to the module locals (module
>> being __main__).  g is not defined remotely, so __main__.g doesn't resolve.
>>  We have some utilities that allow functions to be serialized in a way that
>> works around this, but func.func_defaults is  inspected and wrapped like
>> func itself.  If we added that, then this code would just work, I think.
>>
>> If you avoid passing arguments via the func_defaults, and pass them as
>> arguments explicitly, you avoid the problem:
>>
>> def func(x,f):
>> ...
>>
>> view.map_sync(func, range(10), [f]*10)
>>
>> Will avoid the issue.
>>
>> There is no need to push f before passing it to map.
>>
>> -MinRK
>>
>>
>>>
>>> -Robert
>>>
>>> _______________________________________________
>>> IPython-User mailing list
>>> IPython-User@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>
>>>
>>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120620/81e0aef5/attachment.html