[IPython-User] parallel question

[MinRK]

On Mon, Jan 2, 2012 at 11:11, Toby Burnett <tburnett@uw.edu> wrote:

> Oops: that lambda function does not work with a global variable, my
> specific use case. :-(
>
> This is what does work instead:
>
> @interactive
> def remotefun(f,x):
>    return eval(f,globals())(x)
>
> --Toby Burnett
>
> From: Toby Burnett
> Sent: Monday, January 02, 2012 09:12
> To: 'MinRK'
> Cc: ipython-user@scipy.org
> Subject: RE: [IPython-User] parallel question
>
> I’ve used MinRK’s solution for a while, but wanted to encapsulate the
> scheme into a class that manages a set of engines, and avoid ‘wiring in’
> the function name in code. The tricky thing is to avoid closure, which
> prevents the necessary use of pickle.
> Here is what works, for comments and suggestions.
>
>    def submit(self, code, *pars):
>         """ submit jobs to the engines
>         code : string that must evaluate to a function to be executed on
> the target machine
>         pars : one or more lists of arguments for the function
>         """
>         print ’submitting  %d tasks '%len(pars[0])
>         retun self.lview.map( lambda f,x: eval(f)(x), len(pars[0])*[code],
> *pars))
>
>
> The inelegant feature is duplication of the string defining the function.
>
> --Toby Burnett
>

You might try using the Reference object, for reconstructing remote objects
by name:

from IPython.parallel import Reference
rf = Reference(fname)
# workaround for bug soon to be fixed, preventing References from being
used in map:
rf.__name__ = fname

amr = self.lview.map(rf, *pars)

You can use References to objects by name, so that you can call functions
with arguments without transferring them (See
here<http://minrk.github.com/scipy-tutorial-2011/basic_remote.html?highlight=reference#working-with-the-engine-namespace>
for
some more info).  I had never thought of using References for the callable
itself, but there's no reason not to support it.

Ideally, map should be able to take a Reference as the callable, but
there's a small bug where it is assumed that the function passed to apply
has a `__name__` attribute.  I'll do a PR removing this assumption shortly,
but for now, simply setting the __name__ attribute should allow you to use
this.

This means that you should be able to change your code to:

def submit(self, f, *pars):
    if isinstance(f, basestring):
        name = f
        f = Reference(name)
        f.__name__ = name # workaround
    print ’submitting  %d tasks '%len(pars[0])
    return self.lview.map(f, *pars)

Which should accept *both* locally defined functions and ones that exist
only remotely if given by name.

-MinRK


> From: MinRK [mailto:benjaminrk@gmail.com]
> Sent: Tuesday, December 27, 2011 22:28
> To: Toby Burnett
> Cc: ipython-user@scipy.org
> Subject: Re: [IPython-User] parallel question
>
>
> On Tue, Dec 27, 2011 at 12:36, Toby Burnett <tburnett@uw.edu> wrote:
> Hi,
> With the old MultiEngineClient, I used to set up local objects in the
> individual engines, say a function ‘f’, and then invoke them by name. An
> equivalent thing now is
>                 dview.map_sync(lambda x: eval(‘f’)(x), range(100))
>
> This line works fine, if f is defined as a function on the engines, only
> if I type it on the command line. If the lambda function is precompiled
> code however, I get error messages from the clients about ‘f’ not being
> defined, e.g.
>                 [0:apply]: NameError: name 'f' is not defined
>
> This has me very puzzled, I presume related to how the communication from
> the controller to the engines works?
>
> The lambda in question is defined in a module?  If so, then the issue is
> that the globals() of the lambda is that of its module, instead of the
> user_ns (`__main__` on the engine).  If we didn't do this, calling apply
> with library functions (e.g. os.getpid, or numpy.linalg.norm) would not
> work.  If you are defining a function specifically for interactive use,
> IPython provides a `@interactive` decorator, which makes the function
> behave remotely as if it was defined interactively (when reconstructed on
> the engines, it is constructed with user_ns as globals(), instead of the
> module in which it was defined):
>
> #mymodule
> from IPython.parallel.util import interactive
>
> @interactive
> def foo(x):
>     return eval('f')(x)
>     # can also be just:
>     # return f(x)
>     # as long as there is no `f` defined in the current scope, causing a
> closure to be detected.
>
> ...
>
> In [20]: dview['f'] = lambda x: x*x
> In [21]: dview.map_sync(mymodule.foo, range(100))
>
>
> -MinRK
>
>
>
> --Toby Burnett
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120102/eceefcdf/attachment.html