[IPython-user] processing.Pool in ipython

[Robin]

On Sat, Oct 25, 2008 at 10:48 PM, Robin <robince@gmail.com> wrote:
>
> So using %run is OK - but I still think its a problem, because for
> development and interactive use I would be able to use it from the
> ipython prompt:
>
> ie:
> In [15]: print pool.map(f, range(10))
> Exception in thread Thread-3:
> Traceback (most recent call last):
>  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
> line 460, in __bootstrap
>    self.run()
>  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py",
> line 440, in run
>    self.__target(*self.__args, **self.__kwargs)
>  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/processing-0.52-py2.5-macosx-10.3-fat.egg/processing/pool.py",
> line 207, in _handleTasks
>    put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed

So a bit more googling suggests this error comes from trying to pickle
a lambda function... A bit more investigation then reveals that it
will run fine if the function is defined in an external file:

ie
In [1]: from pooltest import f
In [3]: from processing import Pool
In [4]: pool = Pool(processes=4)
In [5]: pool.map(f,range(10))
Out[5]: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

works,

but if f is defined within ipython:
In [8]: def f(x):
   ...:     return x*x
   ...:
In [10]: pool.map(f,range(10))
the error occurs.

So is there something funny about how ipython handles functions
defined interactively that would explain why they are unpickle-able?

This isn't so serious for my use since I would normally use it with
functions defined in other files, but it would be nice to get the
bottom of it.

Thanks,

Robin