[IPython-user] Ipython and multiprocessing

Robin robince@gmail....
Thu Dec 18 03:15:07 CST 2008


On Thu, Dec 18, 2008 at 1:21 AM, Brian Granger <ellisonbg.net@gmail.com> wrote:
>> The huge deal with multiprocessing is that the spawning of new processes
>> is done by forks under Unix. This is great for efficiency, but also it
>> gives a nice "fealing" to, eg, the parallel loops, as globals are
>> transparently distributed throught the fork. The shared memory
>> abstraction is very nice too. As a result multiprocessing feels really
>> nice for parallele computation on a multi CPU box.
>> These two mechanisms would be a fantastic addition to IPython1, but I
>> suspect that would be a lot of work, and I really don't have time to
>> spend on this.
>
> The great + of multiprocessing (its usage of fork to spawn everything)
> is also its big -.  The fork based model of multiprocessing is
> orthogonal to the type of interactive computing that IPython (and
> python itself) offers.  It also means that the second you want to
> scale beyond a multicore CPU, you have to go back to the drawing
> board.  I think you *could* possibly hook up multiprocessing processes
> across different hosts, but at that point, all the benefits go away
> (true fast shared memory, globals, forking).

I agree that the two things are orthogonal - for me the advantage of
multiprocessing is that I can easily run and script SMP actions on a 8
core box (a great time saver) without having to worry about the
complexities of proper clustering, or having to run multiple
interpreters by hand and collate the results.

> In terms of using IPython with multiprocessing, the core problem is
> that there is no reasonable way of forking an interactive session.  To
> really support interactive parallel computing, experience (our own as
> well as commercial products like Star-P) has shown that you really
> need an architecture that is more like what IPython offers.  It is
> _very_ possible that we could provide multiprocessing-like APIs in
> IPython and we are more than willing to work with folks to develop
> those.

I'm not sure what you mean about no way of forking the interactive
session - it is more or less working now - certainly it works as I
would expect (define everything you need to share before you do the
forks and it seems to work) - the only issue is that ipython doesn't
store interactively defined objects (classes, functions, lamdba
functions) in __main__. If we could get the variables from the current
interactive namespace added to the FakeModule __main__ then I think
everything would work.

>> By the way, I have my own implementation of shared numpy arrays, that I
>> am willing to share if anybody is interested. It is a thin wrapper on the
>> array object of multiprocessing. It works for me, and it would be useful
>> to make it a common object people use with numpy and multiprocessing, but
>> I really don't have time to polish it and consider all the use cases
>> right now. Anybody wants to pick it up?

Have you seen this on the wiki:
http://www.scipy.org/Cookbook/multiprocessing
is it similar to that?


More information about the IPython-user mailing list