[IPython-User] parallel execution with multiprocessing vs. ipcluster

Johann Rohwer jr@sun.ac...
Thu Aug 30 15:10:22 CDT 2012


I'm working on developing a parallel execution framework for our application PySCeS (pysces.sf.net) within the ipcluster framework and 
it works fine.Prompted by the parallel tutorial at EuroScipy I tried to port the code to use the multiprocessing module rather than (or as an alternative to) ipcluster, since this reduces external dependencies - multiprocessing is part of the standard library. 

Interestingly, the test script using multiprocessing works fine when executed within IPython (this is just the plain ipython shell, not ipcluster!), but fails when called with straight python, with the following traceback:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in 
__bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in 
_handle_tasks
    put(task)
PicklingError: Can't pickle <type 'code'>: attribute lookup 
__builtin__.code failed

Before going all the way to construct a minimal example that fails (the instantiated class that gets passed around to the workers is quite complex), I'd like to find out from the list whether there is anything in the IPython code base that might cause this behaviour (i.e. prevent the pickling error above)?  What I did notice is that when I instantiate multiprocessing.Pool() from within IPython, the daughter processes of the workers get spawned as "ipython", whereas when instantiating the pool from within python directly, the worker processes are also plain python of course. Could this cause the difference? I'm a bit stymied as to where to investigate this further.

--Johann



More information about the IPython-User mailing list