[SciPy-user] Multiprocessing, GUIs and IPython

Brian Granger ellisonbg.net@gmail....
Wed Jan 7 14:00:03 CST 2009


Hi,

I see that people are starting to use multiprocessing to parallelize
numerical Python code.  I am wondering if we want to allow/recommend
using multiprocessing in scipy.  Here are some of my concerns:

* Currently multiprocessing doesn't play well with IPython.  Thus, if
scipy starts to use multiprocessing, people will get very unpleasant
surprises when using IPython.  I don't know exactly what the problems
are, but my feeling is that it is unlikely that IPython will ever have
*full* support for multiprocessing.  Some support might be possible,
though.

* I have no idea about how multiprocessing plays with GUIs.  Because
multiprocessing uses fork, my gut feeling is that GUIs would not be
very happy with multiprocessing.  But, I imagine that it really
depends on what exactly multiprocessing does when it forks.  It would
be bad if parts of scipy became unusable from a GUI because of
multiprocessing.

* Multiprocessing doesn't play well with other things as well, such as
Twisted.  Again, if scipy uses multiprocessing, it would become
usuable within Twisted based servers.

What experience have others had with using multiprocessing in these
contexts.  Success?  Failure?  Based on that, what to other people
recommend and think about using multiprocessing in scipy or numpy?  I
guess this also applies to any other project in this realm (sympy,
pymc, ETS, matplotlib, etc., etc.).

Cheers,

Brian


More information about the SciPy-user mailing list