[IPython-User] Slides from my IPython Scipy India 2010 talk

Francesc Alted faltet@pytables....
Tue Dec 14 05:59:44 CST 2010

A Tuesday 14 December 2010 12:01:11 Fernando Perez escrigué:
> > Also, your recent work on IPython/zeromq is amazing.  However, I've
> > always asked myself why you are integrating such a parallel core in
> > IPython.  Would not it be better to detach it and make it a
> > different project?  Which is the logic behind distributing it with
> > IPython?
> Very valid question, but I think the newest codebase finally provides
> a compelling answer in practice.  Since the beginning, we've thought
> (in fact that was the reason Brian and I joined forces years ago)
> that many of the abstractions you want for parallel computing are
> the same that you want for a good interactive system.  If you think
> of either as ultimately manipulating a namespace by sending data to
> it, executing code in it, and analyzing it by retrieving data from
> it, then running code at a console or on a cluster isn't that
> different.
> Unfortunately for a long time we had this idea, but hadn't been able
> to really blend things enough, so the kernels used by the parallel
> ipython 0.10.x code (and prior) were different from the interactive
> shell you type into.  We meant to merge them, but never succeeded in
> doing it.
> Now, however, we're finally getting there: Min's newparallel branch
> (now in the core repo at
> https://github.com/ipython/ipython/tree/newparallel) is already using
> the same zmq abstractions for communication that the Qt console uses,
> and we've cleaned up the core shell code a lot to make it possible to
> use the same object in all kernels.
> Once we review/merge newparallel and finish integrating one single
> core shell object in all incarnations of the kernel, it will be
> possible to debug parallel codes as easily as you work at an
> interactive console (by literally connecting to the real node), for
> example.
> In general, what we now are offering is an object that declares a
> network-accessible API defined in our messaging spec, for
> manipulating a namespace.  We can then put that api to use in
> multiple contexts, reusing one good, common abstraction. 
> Single-user interactive work, multi-user collaborative work and
> parallel computation happen to be different usage patterns of this
> same abstraction.
> Does this seem sensible?

Yes, I suppose so :-)  Frankly, I never realised that an interactive 
console would have so many things in common with a parallel core, but 
apparently it does.

At any rate, my impression continues to be that a split would benefit 
the community.  Those more interested in the interactive work would 
choose traditional IPython (+interfaces for toolkits like GTK or Qt), 
while others may want to take just the parallel code.

Anyway, if such a split is really interesting then I suppose that it 
will eventually happen (in a similar way that the NumPy crew are working 
on splitting parts of the library that can be used by other apps).

Thanks for the detailed explanation!

Francesc Alted

More information about the IPython-User mailing list