[IPython-User] parallel question
Mon Jan 2 13:11:13 CST 2012
Oops: that lambda function does not work with a global variable, my specific use case. :-(
This is what does work instead:
From: Toby Burnett
Sent: Monday, January 02, 2012 09:12
Subject: RE: [IPython-User] parallel question
I’ve used MinRK’s solution for a while, but wanted to encapsulate the scheme into a class that manages a set of engines, and avoid ‘wiring in’ the function name in code. The tricky thing is to avoid closure, which prevents the necessary use of pickle.
Here is what works, for comments and suggestions.
def submit(self, code, *pars):
""" submit jobs to the engines
code : string that must evaluate to a function to be executed on the target machine
pars : one or more lists of arguments for the function
print ’submitting %d tasks '%len(pars)
retun self.lview.map( lambda f,x: eval(f)(x), len(pars)*[code], *pars))
The inelegant feature is duplication of the string defining the function.
From: MinRK [mailto:firstname.lastname@example.org]
Sent: Tuesday, December 27, 2011 22:28
To: Toby Burnett
Subject: Re: [IPython-User] parallel question
On Tue, Dec 27, 2011 at 12:36, Toby Burnett <email@example.com> wrote:
With the old MultiEngineClient, I used to set up local objects in the individual engines, say a function ‘f’, and then invoke them by name. An equivalent thing now is
dview.map_sync(lambda x: eval(‘f’)(x), range(100))
This line works fine, if f is defined as a function on the engines, only if I type it on the command line. If the lambda function is precompiled code however, I get error messages from the clients about ‘f’ not being defined, e.g.
[0:apply]: NameError: name 'f' is not defined
This has me very puzzled, I presume related to how the communication from the controller to the engines works?
The lambda in question is defined in a module? If so, then the issue is that the globals() of the lambda is that of its module, instead of the user_ns (`__main__` on the engine). If we didn't do this, calling apply with library functions (e.g. os.getpid, or numpy.linalg.norm) would not work. If you are defining a function specifically for interactive use, IPython provides a `@interactive` decorator, which makes the function behave remotely as if it was defined interactively (when reconstructed on the engines, it is constructed with user_ns as globals(), instead of the module in which it was defined):
from IPython.parallel.util import interactive
# can also be just:
# return f(x)
# as long as there is no `f` defined in the current scope, causing a closure to be detected.
In : dview['f'] = lambda x: x*x
In : dview.map_sync(mymodule.foo, range(100))
IPython-User mailing list
More information about the IPython-User