[IPython-User] Slides from my IPython Scipy India 2010 talk

Francesc Alted faltet@pytables....
Tue Dec 14 05:59:44 CST 2010


A Tuesday 14 December 2010 12:01:11 Fernando Perez escrigué:
> > Also, your recent work on IPython/zeromq is amazing.  However, I've
> > always asked myself why you are integrating such a parallel core in
> > IPython.  Would not it be better to detach it and make it a
> > different project?  Which is the logic behind distributing it with
> > IPython?
> 
> Very valid question, but I think the newest codebase finally provides
> a compelling answer in practice.  Since the beginning, we've thought
> (in fact that was the reason Brian and I joined forces years ago)
> that many of the abstractions you want for parallel computing are
> the same that you want for a good interactive system.  If you think
> of either as ultimately manipulating a namespace by sending data to
> it, executing code in it, and analyzing it by retrieving data from
> it, then running code at a console or on a cluster isn't that
> different.
> 
> Unfortunately for a long time we had this idea, but hadn't been able
> to really blend things enough, so the kernels used by the parallel
> ipython 0.10.x code (and prior) were different from the interactive
> shell you type into.  We meant to merge them, but never succeeded in
> doing it.
> 
> Now, however, we're finally getting there: Min's newparallel branch
> (now in the core repo at
> https://github.com/ipython/ipython/tree/newparallel) is already using
> the same zmq abstractions for communication that the Qt console uses,
> and we've cleaned up the core shell code a lot to make it possible to
> use the same object in all kernels.
> 
> Once we review/merge newparallel and finish integrating one single
> core shell object in all incarnations of the kernel, it will be
> possible to debug parallel codes as easily as you work at an
> interactive console (by literally connecting to the real node), for
> example.
> 
> In general, what we now are offering is an object that declares a
> network-accessible API defined in our messaging spec, for
> manipulating a namespace.  We can then put that api to use in
> multiple contexts, reusing one good, common abstraction. 
> Single-user interactive work, multi-user collaborative work and
> parallel computation happen to be different usage patterns of this
> same abstraction.
> 
> Does this seem sensible?

Yes, I suppose so :-)  Frankly, I never realised that an interactive 
console would have so many things in common with a parallel core, but 
apparently it does.

At any rate, my impression continues to be that a split would benefit 
the community.  Those more interested in the interactive work would 
choose traditional IPython (+interfaces for toolkits like GTK or Qt), 
while others may want to take just the parallel code.

Anyway, if such a split is really interesting then I suppose that it 
will eventually happen (in a similar way that the NumPy crew are working 
on splitting parts of the library that can be used by other apps).

Thanks for the detailed explanation!

-- 
Francesc Alted


More information about the IPython-User mailing list