[IPython-user] Ipython and multiprocessing

Gael Varoquaux gael.varoquaux@normalesup....
Wed Dec 17 10:49:33 CST 2008


On Wed, Dec 17, 2008 at 08:33:00AM -0800, Brian Granger wrote:
>  We have been doing a lot of work on ipcluster lately
> (merged into trunk, but not released) that will make it much easier to
> start an ipcluster in a wider range on contexts.  Eventually, you will
> be able start an IPython cluster in a similar that you can create a
> Pool in multiprocessing. 

The huge deal with multiprocessing is that the spawning of new processes
is done by forks under Unix. This is great for efficiency, but also it
gives a nice "fealing" to, eg, the parallel loops, as globals are
transparently distributed throught the fork. The shared memory
abstraction is very nice too. As a result multiprocessing feels really
nice for parallele computation on a multi CPU box.
These two mechanisms would be a fantastic addition to IPython1, but I
suspect that would be a lot of work, and I really don't have time to
spend on this.

By the way, I have my own implementation of shared numpy arrays, that I
am willing to share if anybody is interested. It is a thin wrapper on the
array object of multiprocessing. It works for me, and it would be useful
to make it a common object people use with numpy and multiprocessing, but
I really don't have time to polish it and consider all the use cases
right now. Anybody wants to pick it up?

Gaël


More information about the IPython-user mailing list