[IPython-dev] Parallel: Report progress from the engines
Mon Aug 12 18:34:29 CDT 2013
these two notebooks provide examples of monitoring a parallel run (in
this case an MPI one, but you can adapt this to your use case). The
first uses engine-driven data publication, the second uses client-side
polling in a thread; each approach has its pros and cons:
And here is a bit more info on the structure of async results that can
be used for timing/monitoring:
I seem to recall Min had some more examples of this, but can't seem to
find them right now.
On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
> Hi list,
> I'm using the fantastic parallel computing machinery from IPython, which
> works really well. However, the simulations I'm performing take a few
> hours to finish and I'd like to see the progress from each engine. The
> number of tasks is small (usually from five to eight) such that only
> knowing how many tasks have finished is not very useful for me. I need
> to track the progress of each individual task.
> To give an example, with the multiprocessing module I could create a
> Queue that I can pass as argument to the task that will run in the other
> process. My tasks are basically a for loop running a few lines of code
> in each iteration. Therefore, the task can put the iteration number in
> the Queue indicating its progress, which is then read (every 5 seconds)
> from another process that finally prints the progress in the terminal.
> However, I have no idea how I can do this with the IPython parallel
> machinery. Is there any way for the engines to send data back (to the
> controller?) that I can use to track their progress?
> Darlan Cavalcante
> IPython-dev mailing list
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
More information about the IPython-dev