[IPython-User] parallel execution with multiprocessing vs. ipcluster
Thu Aug 30 15:10:22 CDT 2012
I'm working on developing a parallel execution framework for our application PySCeS (pysces.sf.net) within the ipcluster framework and
it works fine.Prompted by the parallel tutorial at EuroScipy I tried to port the code to use the multiprocessing module rather than (or as an alternative to) ipcluster, since this reduces external dependencies - multiprocessing is part of the standard library.
Interestingly, the test script using multiprocessing works fine when executed within IPython (this is just the plain ipython shell, not ipcluster!), but fails when called with straight python, with the following traceback:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in
File "/usr/lib/python2.7/threading.py", line 504, in run
File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in
PicklingError: Can't pickle <type 'code'>: attribute lookup
Before going all the way to construct a minimal example that fails (the instantiated class that gets passed around to the workers is quite complex), I'd like to find out from the list whether there is anything in the IPython code base that might cause this behaviour (i.e. prevent the pickling error above)? What I did notice is that when I instantiate multiprocessing.Pool() from within IPython, the daughter processes of the workers get spawned as "ipython", whereas when instantiating the pool from within python directly, the worker processes are also plain python of course. Could this cause the difference? I'm a bit stymied as to where to investigate this further.
More information about the IPython-User