[IPython-dev] Confusing NameError in Parallelizing with IPython

MinRK benjaminrk@gmail....
Fri Sep 7 18:55:26 CDT 2012


On Fri, Sep 7, 2012 at 2:18 PM, Michael Bilow <michael.k.bilow@gmail.com>wrote:

> Wow, you nailed it, thanks.  I was running ipcluster start in a different
> directory from the one my code was in.  I still don't understand the
> provenance of that particular error, but I can see how it could cause one.


Here's a mor detailed exploration of what is happening:

    import pickle
    from wrapper import parallel_run
    print pickle.dumps(parallel_run)

gives:

    cwrapper
    parallel_run
    p0
    .

In other words, the serialized form of a module function is just the
function name and the module in which it lives.

When the request arrives on the engines, it is deserialized back into its
module namespace like
so<https://github.com/ipython/ipython/blob/master/IPython/utils/pickleutil.py#L98>
:

            try:
                __import__(self.module)
            except ImportError:
                pass
            else:
                g = sys.modules[self.module].__dict__

So what is happening is the import of `wrapper` is failing, but IPython
suppressed that error, and unpacked the function into an empty namespace
instead.  It's possible that this error should *not* be suppressed, because
it certainly would have been more informative to you to see `ImportError:
wrapper` rather than a NameError since it came out in the wrong place.

I don't know what case is actually helped by this suppression, so I am
inclined to just cease catching the
error<https://github.com/ipython/ipython/pull/2395>
.

-MinRK


>
> Thanks!
>
> Michael
>
> On Fri, Sep 7, 2012 at 2:04 PM, MinRK <benjaminrk@gmail.com> wrote:
>
>>
>>
>> On Fri, Sep 7, 2012 at 1:48 PM, Michael Bilow <michael.k.bilow@gmail.com>wrote:
>>
>>> Hi!
>>>
>>> I'm new to this list and to IPython, and I started using IPython to
>>> parallelize some code I'd written (complete code:
>>> https://github.com/chuyelchulo/ipython-nameerror, requires numpy and
>>> PIL). I've been encountering a confusing NameError that keeps popping up
>>> when I try to run this program: The function to be mapped (parallel_run) is
>>> imported from a module different from __main__, but the interpreter
>>> complains that the functions and modules used by parallel_run are not part
>>> of the namespace.
>>>
>>> I asked about the error here (
>>> http://stackoverflow.com/questions/12304847/ipython-parallel-computing-namespace-issues),
>>> and was told to come to this list.
>>>
>>
>> Hi again!
>>
>> When I do the following:
>>
>> $> cd /path/to/ipython-nameerror
>> $> ipcluster start -n 4 --daemon
>> $> python wrapper_wrapper.py
>> Time elapsed:  5.85908985138 s
>>
>> It seems to work as expected.
>>
>> A few quick questions:
>>
>> 1. what is the cwd when you run ipcluster?  Are your engines local or
>> remote?  What do you get from:
>>
>> import os
>> import pprint
>> from IPython import parallel
>>
>> client = parallel.Client()
>> pprint.pprint(client[:].apply_async(os.getcwdu).get_dict())
>>
>> or:
>>
>> %px import wrapper
>>
>> If I start the engines somewhere else such that `wrapper` is not
>> importable, I get the NameErrors you are seeing.
>>
>> -MinRK
>>
>>
>>> Thanks!
>>>
>>> Michael
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-dev/attachments/20120907/db413f9f/attachment.html 


More information about the IPython-dev mailing list