<br><br><div class="gmail_quote">On Mon, Jan 2, 2012 at 11:11, Toby Burnett <span dir="ltr"><<a href="mailto:tburnett@uw.edu">tburnett@uw.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Oops: that lambda function does not work with a global variable, my specific use case. :-(<br>
<br>
This is what does work instead:<br>
<br>
@interactive<br>
def remotefun(f,x):<br>
return eval(f,globals())(x)<br>
<br>
--Toby Burnett<br>
<br>
From: Toby Burnett<br>
Sent: Monday, January 02, 2012 09:12<br>
To: 'MinRK'<br>
Cc: <a href="mailto:ipython-user@scipy.org">ipython-user@scipy.org</a><br>
Subject: RE: [IPython-User] parallel question<br>
<div class="im"><br>
I’ve used MinRK’s solution for a while, but wanted to encapsulate the scheme into a class that manages a set of engines, and avoid ‘wiring in’ the function name in code. The tricky thing is to avoid closure, which prevents the necessary use of pickle.<br>
Here is what works, for comments and suggestions.<br>
<br>
def submit(self, code, *pars):<br>
""" submit jobs to the engines<br>
code : string that must evaluate to a function to be executed on the target machine<br>
pars : one or more lists of arguments for the function<br>
"""<br>
print ’submitting %d tasks '%len(pars[0])<br>
retun self.lview.map( lambda f,x: eval(f)(x), len(pars[0])*[code], *pars))<br>
<br>
<br>
The inelegant feature is duplication of the string defining the function.<br>
<br>
--Toby Burnett<br></div></blockquote><div><br></div><div>You might try using the Reference object, for reconstructing remote objects by name:</div><div><br></div><div>from IPython.parallel import Reference</div><div>rf = Reference(fname)</div>
<div># workaround for bug soon to be fixed, preventing References from being used in map:</div><div>rf.__name__ = fname</div><div><br></div><div>amr = self.lview.map(rf, *pars)</div><div><br></div><div>You can use References to objects by name, so that you can call functions with arguments without transferring them (See <a href="http://minrk.github.com/scipy-tutorial-2011/basic_remote.html?highlight=reference#working-with-the-engine-namespace">here</a> for some more info). I had never thought of using References for the callable itself, but there's no reason not to support it.</div>
<div><br></div><div>Ideally, map should be able to take a Reference as the callable, but there's a small bug where it is assumed that the function passed to apply has a `__name__` attribute. I'll do a PR removing this assumption shortly, but for now, simply setting the __name__ attribute should allow you to use this.</div>
<div><br></div><div>This means that you should be able to change your code to:</div><div><br></div><div>def submit(self, f, *pars):</div><div> if isinstance(f, basestring):</div><div> name = f</div><div> f = Reference(name)</div>
<div> f.__name__ = name # workaround</div><div> print ’submitting %d tasks '%len(pars[0])</div><div> return self.lview.map(f, *pars)</div><div><br></div><div>Which should accept *both* locally defined functions and ones that exist only remotely if given by name.</div>
<div><br></div><div>-MinRK</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
From: MinRK [mailto:<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>]<br>
Sent: Tuesday, December 27, 2011 22:28<br>
To: Toby Burnett<br>
Cc: <a href="mailto:ipython-user@scipy.org">ipython-user@scipy.org</a><br>
Subject: Re: [IPython-User] parallel question<br>
<br>
<br>
</div><div><div></div><div class="h5">On Tue, Dec 27, 2011 at 12:36, Toby Burnett <<a href="mailto:tburnett@uw.edu">tburnett@uw.edu</a>> wrote:<br>
Hi,<br>
With the old MultiEngineClient, I used to set up local objects in the individual engines, say a function ‘f’, and then invoke them by name. An equivalent thing now is<br>
dview.map_sync(lambda x: eval(‘f’)(x), range(100))<br>
<br>
This line works fine, if f is defined as a function on the engines, only if I type it on the command line. If the lambda function is precompiled code however, I get error messages from the clients about ‘f’ not being defined, e.g.<br>
[0:apply]: NameError: name 'f' is not defined<br>
<br>
This has me very puzzled, I presume related to how the communication from the controller to the engines works?<br>
<br>
The lambda in question is defined in a module? If so, then the issue is that the globals() of the lambda is that of its module, instead of the user_ns (`__main__` on the engine). If we didn't do this, calling apply with library functions (e.g. os.getpid, or numpy.linalg.norm) would not work. If you are defining a function specifically for interactive use, IPython provides a `@interactive` decorator, which makes the function behave remotely as if it was defined interactively (when reconstructed on the engines, it is constructed with user_ns as globals(), instead of the module in which it was defined):<br>
<br>
#mymodule<br>
from IPython.parallel.util import interactive<br>
<br>
@interactive<br>
def foo(x):<br>
return eval('f')(x)<br>
# can also be just:<br>
# return f(x)<br>
# as long as there is no `f` defined in the current scope, causing a closure to be detected.<br>
<br>
...<br>
<br>
In [20]: dview['f'] = lambda x: x*x<br>
In [21]: dview.map_sync(mymodule.foo, range(100))<br>
<br>
<br>
-MinRK<br>
<br>
<br>
<br>
--Toby Burnett<br>
<br>
_______________________________________________<br>
IPython-User mailing list<br>
<a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
<br>
</div></div></blockquote></div><br>