[IPython-user] processing.Pool in ipython

Robert Kern robert.kern@gmail....
Sat Oct 25 16:50:58 CDT 2008


Robin wrote:
> On Sat, Oct 25, 2008 at 9:47 PM, Robert Kern <robert.kern@gmail.com> wrote:
>> Robin wrote:
>>> I forgot to add the version information. I on OS X 10.5 with IPython
>>> 0.9.1 on python.org python 2.5.1 with processing module v. 0.52.
>> It might be worth trying the multiprocessing package, instead. During the
>> process of getting pyprocessing into Python 2.6, some bugs were fixed. You can
>> get the backported version for Python 2.5 here:
>>
>>   http://code.google.com/p/python-multiprocessing/
> 
> Actually I did try with multiprocessing after posting to the list, but
> had the same result with ipython - and even worse luck, in that even
> with normal python the same script above prints the correct output but
> then doesn't exit cleanly. So I'm sticking to processing for now.
> 
>> In any case, it's not so much of a problem with mixing processing and IPython
>> but rather that running scripts via "ipython pool_test.py" is not created the
>> right namespaces for pickle to be able to reload it in the subprocesses. I'm not
>> sure if this is really a bug or just an unintended but expected consequence of
>> some other decision. "ipython foo.py" is *not* intended to replicate "python
>> foo.py" exactly.
>>
>> The example does work when using "%run pool_test.py" which should cover almost
>> all use cases.
> 
> Actually I hadn't tried with %run - that does work, but typing the
> commands in to the ipython prompt gives the same error. It was this
> interactive use that alerted me to the problem originally - the
> ipython foo.py was just a quick way to demonstrate the problem and for
> me to test without typing the commands in - its not something I do
> often.
> 
> So using %run is OK - but I still think its a problem, because for
> development and interactive use I would be able to use it from the
> ipython prompt:
> 
> ie:
> In [15]: print pool.map(f, range(10))
> Exception in thread Thread-3:
> Traceback (most recent call last):
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
> line 460, in __bootstrap
>     self.run()
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
> line 440, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/processing-0.52-py2.5-macosx-10.3-fat.egg/processing/pool.py",
> line 207, in _handleTasks
>     put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed

Yes, you're right. Hmm.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



More information about the IPython-user mailing list