[IPython-User] parallel execution with multiprocessing vs. ipcluster

Johann Rohwer jr@sun.ac...
Mon Sep 3 04:52:49 CDT 2012


To follow up on my own post:

On Thursday 30 August 2012 22:10:22 Johann Rohwer wrote:
> Interestingly, the test script using multiprocessing works fine when
> executed within IPython (this is just the plain ipython shell, not
> ipcluster!), but fails when called with straight python, with the
> following traceback:
> 
> Exception in thread Thread-2:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 551, in
> __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.7/threading.py", line 504, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in
> _handle_tasks
>     put(task)
> PicklingError: Can't pickle <type 'code'>: attribute lookup
> __builtin__.code failed

It turns out I needed the codeutil module that is shipped in 
IPython.utils in order to pickle code objects. Obviously ipython loads 
this by default during its startup, but it is not imported in the 
regular python interpreter. Once I copied that file into my own 
application and imported it in its __init__.py, then everything works 
fine!

--Johann
E-pos vrywaringsklousule

Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie.

E-mail disclaimer

This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.


More information about the IPython-User mailing list