[IPython-User] parallel question

MinRK benjaminrk@gmail....
Mon Jan 2 17:49:25 CST 2012


PR in question <https://github.com/ipython/ipython/pull/1222>.

On Mon, Jan 2, 2012 at 15:46, MinRK <benjaminrk@gmail.com> wrote:

>
>
> On Mon, Jan 2, 2012 at 11:11, Toby Burnett <tburnett@uw.edu> wrote:
>
>> Oops: that lambda function does not work with a global variable, my
>> specific use case. :-(
>>
>> This is what does work instead:
>>
>> @interactive
>> def remotefun(f,x):
>>    return eval(f,globals())(x)
>>
>> --Toby Burnett
>>
>> From: Toby Burnett
>> Sent: Monday, January 02, 2012 09:12
>> To: 'MinRK'
>> Cc: ipython-user@scipy.org
>> Subject: RE: [IPython-User] parallel question
>>
>> I’ve used MinRK’s solution for a while, but wanted to encapsulate the
>> scheme into a class that manages a set of engines, and avoid ‘wiring in’
>> the function name in code. The tricky thing is to avoid closure, which
>> prevents the necessary use of pickle.
>> Here is what works, for comments and suggestions.
>>
>>    def submit(self, code, *pars):
>>         """ submit jobs to the engines
>>         code : string that must evaluate to a function to be executed on
>> the target machine
>>         pars : one or more lists of arguments for the function
>>         """
>>         print ’submitting  %d tasks '%len(pars[0])
>>         retun self.lview.map( lambda f,x: eval(f)(x),
>> len(pars[0])*[code], *pars))
>>
>>
>> The inelegant feature is duplication of the string defining the function.
>>
>> --Toby Burnett
>>
>
> You might try using the Reference object, for reconstructing remote
> objects by name:
>
> from IPython.parallel import Reference
> rf = Reference(fname)
> # workaround for bug soon to be fixed, preventing References from being
> used in map:
> rf.__name__ = fname
>
> amr = self.lview.map(rf, *pars)
>
> You can use References to objects by name, so that you can call functions
> with arguments without transferring them (See here<http://minrk.github.com/scipy-tutorial-2011/basic_remote.html?highlight=reference#working-with-the-engine-namespace> for
> some more info).  I had never thought of using References for the callable
> itself, but there's no reason not to support it.
>
> Ideally, map should be able to take a Reference as the callable, but
> there's a small bug where it is assumed that the function passed to apply
> has a `__name__` attribute.  I'll do a PR removing this assumption shortly,
> but for now, simply setting the __name__ attribute should allow you to use
> this.
>
> This means that you should be able to change your code to:
>
> def submit(self, f, *pars):
>     if isinstance(f, basestring):
>         name = f
>         f = Reference(name)
>         f.__name__ = name # workaround
>     print ’submitting  %d tasks '%len(pars[0])
>     return self.lview.map(f, *pars)
>
> Which should accept *both* locally defined functions and ones that exist
> only remotely if given by name.
>
> -MinRK
>
>
>> From: MinRK [mailto:benjaminrk@gmail.com]
>> Sent: Tuesday, December 27, 2011 22:28
>> To: Toby Burnett
>> Cc: ipython-user@scipy.org
>> Subject: Re: [IPython-User] parallel question
>>
>>
>> On Tue, Dec 27, 2011 at 12:36, Toby Burnett <tburnett@uw.edu> wrote:
>> Hi,
>> With the old MultiEngineClient, I used to set up local objects in the
>> individual engines, say a function ‘f’, and then invoke them by name. An
>> equivalent thing now is
>>                 dview.map_sync(lambda x: eval(‘f’)(x), range(100))
>>
>> This line works fine, if f is defined as a function on the engines, only
>> if I type it on the command line. If the lambda function is precompiled
>> code however, I get error messages from the clients about ‘f’ not being
>> defined, e.g.
>>                 [0:apply]: NameError: name 'f' is not defined
>>
>> This has me very puzzled, I presume related to how the communication from
>> the controller to the engines works?
>>
>> The lambda in question is defined in a module?  If so, then the issue is
>> that the globals() of the lambda is that of its module, instead of the
>> user_ns (`__main__` on the engine).  If we didn't do this, calling apply
>> with library functions (e.g. os.getpid, or numpy.linalg.norm) would not
>> work.  If you are defining a function specifically for interactive use,
>> IPython provides a `@interactive` decorator, which makes the function
>> behave remotely as if it was defined interactively (when reconstructed on
>> the engines, it is constructed with user_ns as globals(), instead of the
>> module in which it was defined):
>>
>> #mymodule
>> from IPython.parallel.util import interactive
>>
>> @interactive
>> def foo(x):
>>     return eval('f')(x)
>>     # can also be just:
>>     # return f(x)
>>     # as long as there is no `f` defined in the current scope, causing a
>> closure to be detected.
>>
>> ...
>>
>> In [20]: dview['f'] = lambda x: x*x
>> In [21]: dview.map_sync(mymodule.foo, range(100))
>>
>>
>> -MinRK
>>
>>
>>
>> --Toby Burnett
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120102/ce94e6f2/attachment-0001.html 


More information about the IPython-User mailing list