[IPython-User] Migrating to ipcluster

Mario Ceresa mrceresa@gmail....
Mon Feb 14 12:16:22 CST 2011


Hello everybody,
first of all thanks a lot for all your hard work on ipython. It is a
wonderful project! I'm a PhD student in medical imaging and I
routinely use  it to do all my lab works.

After recently discovering its parallel processing capabilities, I
threw my own cluster management code away and replaced it with
ipcluster: it worked like a charm!

However, I'm not sure how to:

* Get the the failed and succeeded counters increment properly in the
TaskClient queue_status. For me they are always zero, no matters what
happens to the jobs:
(e.g.)
In [7]: tc.queue_status()
Out[7]: {'failed': 0, 'pending': 12, 'scheduled': 63, 'succeeded': 0}

* I'd like to see the output of the code which is executed on the
nodes. There are no files in the ipython log dir and the tty only
shows the "engine registration succeeded" message.

* I noticed that if I send too much jobs to the cluster (> 100 ) the
nodes receive each only a jobs and then the ipython session hangs. The
workers will finish the job that they receive, but every further tries
to contact the cluster fail with a TimeoutException.

* Is it possible to set up a callback so that when a task completes  a
user function is called?

I searched the documentation thoroughly and even peeked a bit at the
code, but without success.

Thanks for any hints,

With best regard,

Mario


More information about the IPython-User mailing list