[IPython-dev] Parallel map

Gael Varoquaux gael.varoquaux@normalesup....
Sat Mar 8 04:41:13 CST 2008


On Sat, Mar 08, 2008 at 02:28:02AM -0800, Fernando Perez wrote:
> We need to summarize the recent work done at the sage/scipy/ipython
> sprint.

I did see some recent commits on Trac :->.

> In particular, Min did a lot of excellent work *precisely* on
> this issue, most (if not all) of which is already committed, to
> provide a full ipython daemon for process control.  This allows you to
> do exactly that, to create/control/destroy engines and/or controllers
> from within python scripts.

Great. Now a very important question: can the daemon be started from
Python without an os.system apparent for the user? I want to be able to
log in a box, and run a pPthon script that does parallel computing with
only a few lines of Python code.

> The good thing is that you're already a bzr launchpad team member, so
> I'm sure you'll soon be contributing this code and docs :)

Hum. Currently I am physicaly killing myself doing way to many things.
No, seriously, its worst than it has ever been, and I have a dangerously
off balance sleep pattern. Maybe I'll get better next year when I start
my new job. Maybe I'll even use ipython1 as a full part of my day work.

> We finally figured out a trick using the 'with' statement to allow you
> to write code like

> with all_engines:
>   do_in_remote_engines(..)

That's really nice. Now a big question is, does that remove the need for
all the objects in the parallel code to be picklable. Because that is
what was limiting me here.

Cheers,

Gaël



More information about the IPython-dev mailing list