[IPython-User] Using parallel tools in notebook

Michael Waskom mwaskom@stanford....
Fri Jan 27 20:34:51 CST 2012

Hi Min,

Thanks for clearing this up.

Just for email archive purposes, I'm pretty sure you meant:

dv = rc[:]
dv["inner_func"] = inner_func

Thanks again, things are running smoothly now!


On Wed, Jan 25, 2012 at 2:39 PM, MinRK <benjaminrk@gmail.com> wrote:
> On Wed, Jan 25, 2012 at 11:55, Michael Waskom <mwaskom@stanford.edu> wrote:
>> Hi,
>> I have very little experience using the parallel tools in IPython
>> directly, but I tried basically the simplest example the other day and
>> was quite pleased.  Now I would like to use it in the context of the
>> notebook, and it seems the simple approach does not work as expected.
>> Here's a trivial notebook session demonstrating my problem::
>> from IPython.parallel import Client, error
>> def inner_func(a):
>>     return a
>> def to_be_mapped(b):
>>     return inner_func(b)
>> rc = Client()
>> If I try to do::
>> print rc[:].map_sync(inner_func, range(5))
>> It prints:
>> [0, 1, 2, 3, 4]
>> as expected.
>> If I try, however:
>> print rc[:].map_sync(to_be_mapped, range(5))
>> It immediately raises an exception:
>> CompositeError: one or more exceptions from call to method: to_be_mapped
>> [0:apply]: NameError: global name 'inner_func' is not defined
>> [1:apply]: NameError: global name 'inner_func' is not defined
>> [2:apply]: NameError: global name 'inner_func' is not defined
>> [3:apply]: NameError: global name 'inner_func' is not defined
>> [4:apply]: NameError: global name 'inner_func' is not defined
>> Obviously there's some namespace issue here, but there doesn't seem to
>> be any specific information in the docs about using the notebook and
>> parallel tools together, so I'm wondering if there is a simple answer
>> that doesn't require me to fully digest all of the parallel mechanics.
> There's nothing notebook-specific here, it's just a matter of scope.
> In `to_be_mapped`, `inner_func` is *just a name*.  The one
> IPython-specific bit is that if you ran all of that code in a script
> (or even in a single cell), it would identify the reference to
> inner_func in to_be_mapped as a closure, and balk.  This doesn't
> happen in the notebook because cells are compiled separately, so you
> don't get cross-cell closures.
> When `to_be_mapped` is called on the engine, it looks for `inner_func`
> from globals(), which is the namespace on the engine, and inner_func
> is simply not there.  You can push your local function to the engines
> with:
>    rc['inner_func'] = inner_func
> at which point your code should do what you appear to expect.
> To sum up: Do not assume that names you have defined locally are
> available on engines unless you have explicitly sent them with `push`
> or defined them with `execute`.
> -MinRK
>> Thanks!
>> Michael
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user

More information about the IPython-User mailing list