[IPython-user] processing.Pool in ipython

Robin robince@gmail....
Sat Oct 25 16:48:36 CDT 2008


On Sat, Oct 25, 2008 at 9:47 PM, Robert Kern <robert.kern@gmail.com> wrote:
> Robin wrote:
>> I forgot to add the version information. I on OS X 10.5 with IPython
>> 0.9.1 on python.org python 2.5.1 with processing module v. 0.52.
>
> It might be worth trying the multiprocessing package, instead. During the
> process of getting pyprocessing into Python 2.6, some bugs were fixed. You can
> get the backported version for Python 2.5 here:
>
>   http://code.google.com/p/python-multiprocessing/

Actually I did try with multiprocessing after posting to the list, but
had the same result with ipython - and even worse luck, in that even
with normal python the same script above prints the correct output but
then doesn't exit cleanly. So I'm sticking to processing for now.

>
> In any case, it's not so much of a problem with mixing processing and IPython
> but rather that running scripts via "ipython pool_test.py" is not created the
> right namespaces for pickle to be able to reload it in the subprocesses. I'm not
> sure if this is really a bug or just an unintended but expected consequence of
> some other decision. "ipython foo.py" is *not* intended to replicate "python
> foo.py" exactly.
>
> The example does work when using "%run pool_test.py" which should cover almost
> all use cases.

Actually I hadn't tried with %run - that does work, but typing the
commands in to the ipython prompt gives the same error. It was this
interactive use that alerted me to the problem originally - the
ipython foo.py was just a quick way to demonstrate the problem and for
me to test without typing the commands in - its not something I do
often.

So using %run is OK - but I still think its a problem, because for
development and interactive use I would be able to use it from the
ipython prompt:

ie:
In [15]: print pool.map(f, range(10))
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
line 460, in __bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
line 440, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/processing-0.52-py2.5-macosx-10.3-fat.egg/processing/pool.py",
line 207, in _handleTasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

Thanks very much for you help,

Robin


More information about the IPython-user mailing list