[IPython-User] Using parallel tools in notebook
Fri Jan 27 20:34:51 CST 2012
Thanks for clearing this up.
Just for email archive purposes, I'm pretty sure you meant:
dv = rc[:]
dv["inner_func"] = inner_func
Thanks again, things are running smoothly now!
On Wed, Jan 25, 2012 at 2:39 PM, MinRK <email@example.com> wrote:
> On Wed, Jan 25, 2012 at 11:55, Michael Waskom <firstname.lastname@example.org> wrote:
>> I have very little experience using the parallel tools in IPython
>> directly, but I tried basically the simplest example the other day and
>> was quite pleased. Now I would like to use it in the context of the
>> notebook, and it seems the simple approach does not work as expected.
>> Here's a trivial notebook session demonstrating my problem::
>> from IPython.parallel import Client, error
>> def inner_func(a):
>> return a
>> def to_be_mapped(b):
>> return inner_func(b)
>> rc = Client()
>> If I try to do::
>> print rc[:].map_sync(inner_func, range(5))
>> It prints:
>> [0, 1, 2, 3, 4]
>> as expected.
>> If I try, however:
>> print rc[:].map_sync(to_be_mapped, range(5))
>> It immediately raises an exception:
>> CompositeError: one or more exceptions from call to method: to_be_mapped
>> [0:apply]: NameError: global name 'inner_func' is not defined
>> [1:apply]: NameError: global name 'inner_func' is not defined
>> [2:apply]: NameError: global name 'inner_func' is not defined
>> [3:apply]: NameError: global name 'inner_func' is not defined
>> [4:apply]: NameError: global name 'inner_func' is not defined
>> Obviously there's some namespace issue here, but there doesn't seem to
>> be any specific information in the docs about using the notebook and
>> parallel tools together, so I'm wondering if there is a simple answer
>> that doesn't require me to fully digest all of the parallel mechanics.
> There's nothing notebook-specific here, it's just a matter of scope.
> In `to_be_mapped`, `inner_func` is *just a name*. The one
> IPython-specific bit is that if you ran all of that code in a script
> (or even in a single cell), it would identify the reference to
> inner_func in to_be_mapped as a closure, and balk. This doesn't
> happen in the notebook because cells are compiled separately, so you
> don't get cross-cell closures.
> When `to_be_mapped` is called on the engine, it looks for `inner_func`
> from globals(), which is the namespace on the engine, and inner_func
> is simply not there. You can push your local function to the engines
> rc['inner_func'] = inner_func
> at which point your code should do what you appear to expect.
> To sum up: Do not assume that names you have defined locally are
> available on engines unless you have explicitly sent them with `push`
> or defined them with `execute`.
>> IPython-User mailing list
More information about the IPython-User