[IPython-User] parallel question

[Toby Burnett]

Oops: that lambda function does not work with a global variable, my specific use case. :-(

This is what does work instead:

@interactive
def remotefun(f,x):
    return eval(f,globals())(x) 

--Toby Burnett

From: Toby Burnett 
Sent: Monday, January 02, 2012 09:12
To: 'MinRK'
Cc: ipython-user@scipy.org
Subject: RE: [IPython-User] parallel question

I’ve used MinRK’s solution for a while, but wanted to encapsulate the scheme into a class that manages a set of engines, and avoid ‘wiring in’ the function name in code. The tricky thing is to avoid closure, which prevents the necessary use of pickle.
Here is what works, for comments and suggestions.

   def submit(self, code, *pars):
        """ submit jobs to the engines
        code : string that must evaluate to a function to be executed on the target machine
        pars : one or more lists of arguments for the function
        """
        print ’submitting  %d tasks '%len(pars[0])
        retun self.lview.map( lambda f,x: eval(f)(x), len(pars[0])*[code], *pars))


The inelegant feature is duplication of the string defining the function. 

--Toby Burnett

From: MinRK [mailto:benjaminrk@gmail.com] 
Sent: Tuesday, December 27, 2011 22:28
To: Toby Burnett
Cc: ipython-user@scipy.org
Subject: Re: [IPython-User] parallel question


On Tue, Dec 27, 2011 at 12:36, Toby Burnett <tburnett@uw.edu> wrote:
Hi,
With the old MultiEngineClient, I used to set up local objects in the individual engines, say a function ‘f’, and then invoke them by name. An equivalent thing now is 
                dview.map_sync(lambda x: eval(‘f’)(x), range(100))
 
This line works fine, if f is defined as a function on the engines, only if I type it on the command line. If the lambda function is precompiled code however, I get error messages from the clients about ‘f’ not being defined, e.g.
                [0:apply]: NameError: name 'f' is not defined
 
This has me very puzzled, I presume related to how the communication from the controller to the engines works?

The lambda in question is defined in a module?  If so, then the issue is that the globals() of the lambda is that of its module, instead of the user_ns (`__main__` on the engine).  If we didn't do this, calling apply with library functions (e.g. os.getpid, or numpy.linalg.norm) would not work.  If you are defining a function specifically for interactive use, IPython provides a `@interactive` decorator, which makes the function behave remotely as if it was defined interactively (when reconstructed on the engines, it is constructed with user_ns as globals(), instead of the module in which it was defined):

#mymodule
from IPython.parallel.util import interactive

@interactive
def foo(x):
    return eval('f')(x)
    # can also be just:
    # return f(x)
    # as long as there is no `f` defined in the current scope, causing a closure to be detected.

...

In [20]: dview['f'] = lambda x: x*x
In [21]: dview.map_sync(mymodule.foo, range(100))
 

-MinRK

 
 
--Toby Burnett

_______________________________________________
IPython-User mailing list
IPython-User@scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user