[IPython-User] Using parallel tools in notebook

MinRK benjaminrk@gmail....
Wed Jan 25 16:39:30 CST 2012


On Wed, Jan 25, 2012 at 11:55, Michael Waskom <mwaskom@stanford.edu> wrote:
> Hi,
>
> I have very little experience using the parallel tools in IPython
> directly, but I tried basically the simplest example the other day and
> was quite pleased.  Now I would like to use it in the context of the
> notebook, and it seems the simple approach does not work as expected.
>
> Here's a trivial notebook session demonstrating my problem::
>
> from IPython.parallel import Client, error
> def inner_func(a):
>     return a
> def to_be_mapped(b):
>     return inner_func(b)
> rc = Client()
>
> If I try to do::
>
> print rc[:].map_sync(inner_func, range(5))
>
> It prints:
>
> [0, 1, 2, 3, 4]
>
> as expected.
>
> If I try, however:
>
> print rc[:].map_sync(to_be_mapped, range(5))
>
> It immediately raises an exception:
>
> CompositeError: one or more exceptions from call to method: to_be_mapped
> [0:apply]: NameError: global name 'inner_func' is not defined
> [1:apply]: NameError: global name 'inner_func' is not defined
> [2:apply]: NameError: global name 'inner_func' is not defined
> [3:apply]: NameError: global name 'inner_func' is not defined
> [4:apply]: NameError: global name 'inner_func' is not defined
>
> Obviously there's some namespace issue here, but there doesn't seem to
> be any specific information in the docs about using the notebook and
> parallel tools together, so I'm wondering if there is a simple answer
> that doesn't require me to fully digest all of the parallel mechanics.

There's nothing notebook-specific here, it's just a matter of scope.
In `to_be_mapped`, `inner_func` is *just a name*.  The one
IPython-specific bit is that if you ran all of that code in a script
(or even in a single cell), it would identify the reference to
inner_func in to_be_mapped as a closure, and balk.  This doesn't
happen in the notebook because cells are compiled separately, so you
don't get cross-cell closures.

When `to_be_mapped` is called on the engine, it looks for `inner_func`
from globals(), which is the namespace on the engine, and inner_func
is simply not there.  You can push your local function to the engines
with:

    rc['inner_func'] = inner_func

at which point your code should do what you appear to expect.

To sum up: Do not assume that names you have defined locally are
available on engines unless you have explicitly sent them with `push`
or defined them with `execute`.

-MinRK

>
> Thanks!
>
> Michael
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user


More information about the IPython-User mailing list