[IPython-User] parallel question

Toby Burnett tburnett@uw....
Mon Jan 2 13:11:13 CST 2012

Oops: that lambda function does not work with a global variable, my specific use case. :-(

This is what does work instead:

def remotefun(f,x):
    return eval(f,globals())(x) 

--Toby Burnett

From: Toby Burnett 
Sent: Monday, January 02, 2012 09:12
To: 'MinRK'
Cc: ipython-user@scipy.org
Subject: RE: [IPython-User] parallel question

I’ve used MinRK’s solution for a while, but wanted to encapsulate the scheme into a class that manages a set of engines, and avoid ‘wiring in’ the function name in code. The tricky thing is to avoid closure, which prevents the necessary use of pickle.
Here is what works, for comments and suggestions.

   def submit(self, code, *pars):
        """ submit jobs to the engines
        code : string that must evaluate to a function to be executed on the target machine
        pars : one or more lists of arguments for the function
        print ’submitting  %d tasks '%len(pars[0])
        retun self.lview.map( lambda f,x: eval(f)(x), len(pars[0])*[code], *pars))

The inelegant feature is duplication of the string defining the function. 

--Toby Burnett

From: MinRK [mailto:benjaminrk@gmail.com] 
Sent: Tuesday, December 27, 2011 22:28
To: Toby Burnett
Cc: ipython-user@scipy.org
Subject: Re: [IPython-User] parallel question

On Tue, Dec 27, 2011 at 12:36, Toby Burnett <tburnett@uw.edu> wrote:
With the old MultiEngineClient, I used to set up local objects in the individual engines, say a function ‘f’, and then invoke them by name. An equivalent thing now is 
                dview.map_sync(lambda x: eval(‘f’)(x), range(100))
This line works fine, if f is defined as a function on the engines, only if I type it on the command line. If the lambda function is precompiled code however, I get error messages from the clients about ‘f’ not being defined, e.g.
                [0:apply]: NameError: name 'f' is not defined
This has me very puzzled, I presume related to how the communication from the controller to the engines works?

The lambda in question is defined in a module?  If so, then the issue is that the globals() of the lambda is that of its module, instead of the user_ns (`__main__` on the engine).  If we didn't do this, calling apply with library functions (e.g. os.getpid, or numpy.linalg.norm) would not work.  If you are defining a function specifically for interactive use, IPython provides a `@interactive` decorator, which makes the function behave remotely as if it was defined interactively (when reconstructed on the engines, it is constructed with user_ns as globals(), instead of the module in which it was defined):

from IPython.parallel.util import interactive

def foo(x):
    return eval('f')(x)
    # can also be just:
    # return f(x)
    # as long as there is no `f` defined in the current scope, causing a closure to be detected.


In [20]: dview['f'] = lambda x: x*x
In [21]: dview.map_sync(mymodule.foo, range(100))


--Toby Burnett

IPython-User mailing list

More information about the IPython-User mailing list