[IPython-user] Ipython and multiprocessing

Robin robince@gmail....
Wed Dec 17 05:57:18 CST 2008


On Wed, Dec 17, 2008 at 11:33 AM, Michael Schmuker <schmuker@gmail.com> wrote:
> The strange thing is that it works within plain python, but not in
> IPython. So the multiprocessing module seems to be able to pickle
> interactively defined functions wuite well, but it seems IPython is
> doing something to Pickle which prevents this mechanism form working.

IPython isn't doing anything to pickle, but it does funny things to
the namespace. In plain python, an interactively defined function
pickles as __main__.function which is available in child processes
(pickle doesnt pickle code, it just saves references to existing
modules). As mentioned in those threads, IPython does lots of magic to
the namespace, including replacing __main__ with a FakeModule
instance, which doesn't contain the interactively defined functions,
hence the problem.

> IMHO multiprocessing would perfectly blend in with IPython's parallel
> computing properties, especially for trivial parallelization on SMP
> machines, because one does not need to start and keep track of an
> ipcluster.

I agree but from the other threads it seems the problem involved the
real internals of IPython and might be difficult to solve.

Robin


More information about the IPython-user mailing list