[IPython-User] Slides from my IPython Scipy India 2010 talk

Fernando Perez fperez.net@gmail....
Tue Dec 14 05:01:11 CST 2010


Hi Francesc,

On Tue, Dec 14, 2010 at 3:01 PM, Francesc Alted <faltet@pytables.org> wrote:
> Hi Fernando,
>
> Excellent talk!  It is nice to see how you are always plenty of ideas
> (and still have the enthusiasm to undertake them).

Thanks for the kind words!  Though in reality, it's easy to be
enthusiastic when you have a team as amazing as we have here, building
these tools.  This year has really been a lot of fun.

> Also, your recent work on IPython/zeromq is amazing.  However, I've
> always asked myself why you are integrating such a parallel core in
> IPython.  Would not it be better to detach it and make it a different
> project?  Which is the logic behind distributing it with IPython?

Very valid question, but I think the newest codebase finally provides
a compelling answer in practice.  Since the beginning, we've thought
(in fact that was the reason Brian and I joined forces years ago) that
many of the abstractions you want for parallel computing are the same
that you want for a good interactive system.  If you think of either
as ultimately manipulating a namespace by sending data to it,
executing code in it, and analyzing it by retrieving data from it,
then running code at a console or on a cluster isn't that different.

Unfortunately for a long time we had this idea, but hadn't been able
to really blend things enough, so the kernels used by the parallel
ipython 0.10.x code (and prior) were different from the interactive
shell you type into.  We meant to merge them, but never succeeded in
doing it.

Now, however, we're finally getting there: Min's newparallel branch
(now in the core repo at
https://github.com/ipython/ipython/tree/newparallel) is already using
the same zmq abstractions for communication that the Qt console uses,
and we've cleaned up the core shell code a lot to make it possible to
use the same object in all kernels.

Once we review/merge newparallel and finish integrating one single
core shell object in all incarnations of the kernel, it will be
possible to debug parallel codes as easily as you work at an
interactive console (by literally connecting to the real node), for
example.

In general, what we now are offering is an object that declares a
network-accessible API defined in our messaging spec, for manipulating
a namespace.  We can then put that api to use in multiple contexts,
reusing one good, common abstraction.  Single-user interactive work,
multi-user collaborative work and parallel computation happen to be
different usage patterns of this same abstraction.

Does this seem sensible?

Regards,


f


More information about the IPython-User mailing list