[IPython-User] Problem with load_balanced_view()

[MinRK]

On Fri, Jun 15, 2012 at 7:54 AM, Neal Becker <ndbecker2@gmail.com> wrote:

> What am I doing wrong here?  Tried to call 'run_line' directly, but I get
> 'run_line' not defined.  So I tried again with decorator, but no diff:
>
> from my_script import run_line
>
> from IPython.parallel import Client
>
> rc = Client()
> lview = rc.load_balanced_view()
>
> if __name__ == '__main__':
>
> [ build list of tests...]
>
>    @lview.parallel()
>    def run_it (cmdline):
>        return run_line (cmdline)
>
>    parallel_result = lview.map(lambda t: run_it(cmdline=t), tests)
>
> CompositeError: one or more exceptions from call to method: <lambda>
> [4:apply]: NameError: global name 'run_it' is not defined
> [1:apply]: NameError: global name 'run_it' is not defined
> [0:apply]: NameError: global name 'run_it' is not defined
> [2:apply]: NameError: global name 'run_it' is not defined
> [7:apply]: NameError: global name 'run_it' is not defined
> [3:apply]: NameError: global name 'run_it' is not defined
>


The parallel decorator makes a function that will run remotely, so you
would do:

@view.parallel()
def run_it(cmdline):
    # import might be needed here for namespace reasons
    from my_script import run_line
    run_line(cmdline)

run_it.map(tests)


The decorated run_it function *itself* executes remotely, so you wouldn't
want to pass it to apply/map.

If you want to use view.apply/map, then you don't need to make
parallel/remote functions with the decorators.

Also note that name resolution inside interactively defined functions is
based on the *engine* globals, not the client's,
so the following:

import string

def fetch_lower():
    return string.lowercase

view.apply_sync(fetch_lower)

will raise NameError on string

There are (at least) four ways to deal with imports on engines:

You can sync local/remote imports (this works in most cases, but not
'import foo as bar'):

with view.sync_imports():
    import string

just execute the remote import statement:

view.execute("import string")

decorate functions with `@require` for the modules they need:

from IPython.parallel import require

@require('string') # can pass name *or* module
def fetch_lower():
    return string.lowercase

or (this is the one I used most often) just put imports inside your
interactively defined functions:

def fetch_lower():
    import string
    return string.lowercase


-MinRK


>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120615/fc0a77ba/attachment-0001.html