[IPython-dev] Parallel: Report progress from the engines
Darlan Cavalcante Moreira
Tue Aug 13 22:07:20 CDT 2013
Thank you Fernando and MinRK.
I haven't used MPI before. I'll need some time to dig through these
examples, but it's a good start.
I tried to run the notebooks, but I had a problem with the line below
from IPython.kernel.zmq.datapub import publish_data
which gives me an import error, since IPython.kernel is not used
anymore. I could not find out where this function was moved to. Googling
for publish_data I found the notebook below
which confirms this function is exactly what I was looking for (as long
as I can get the value of the published variable while the task is still
running). Is it still in IPython?
For now MinRK suggestion seems like a good workaround. I'll have to
change my code to use asynchronous tasks instead of sync tasks as I do
now, but that is easy enough.
> You can also monitor the progress with simple print statements by viewing
> ar.stdout <https://gist.github.com/minrk/6171348>
> On Mon, Aug 12, 2013 at 4:34 PM, Fernando Perez <email@example.com>wrote:
>> these two notebooks provide examples of monitoring a parallel run (in
>> this case an MPI one, but you can adapt this to your use case). The
>> first uses engine-driven data publication, the second uses client-side
>> polling in a thread; each approach has its pros and cons:
>> And here is a bit more info on the structure of async results that can
>> be used for timing/monitoring:
>> I seem to recall Min had some more examples of this, but can't seem to
>> find them right now.
>> On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
>> <firstname.lastname@example.org> wrote:
>> > Hi list,
>> > I'm using the fantastic parallel computing machinery from IPython, which
>> > works really well. However, the simulations I'm performing take a few
>> > hours to finish and I'd like to see the progress from each engine. The
>> > number of tasks is small (usually from five to eight) such that only
>> > knowing how many tasks have finished is not very useful for me. I need
>> > to track the progress of each individual task.
>> > To give an example, with the multiprocessing module I could create a
>> > Queue that I can pass as argument to the task that will run in the other
>> > process. My tasks are basically a for loop running a few lines of code
>> > in each iteration. Therefore, the task can put the iteration number in
>> > the Queue indicating its progress, which is then read (every 5 seconds)
>> > from another process that finally prints the progress in the terminal.
>> > However, I have no idea how I can do this with the IPython parallel
>> > machinery. Is there any way for the engines to send data back (to the
>> > controller?) that I can use to track their progress?
>> > --
>> > Darlan Cavalcante
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPythonemail@example.com
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> Fernando Perez (@fperez_org; http://fperez.org)
>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>> fernando.perez-at-berkeley: contact me here for any direct mail
>> IPython-dev mailing list
> IPython-dev mailing list
Darlan Cavalcante Moreira
More information about the IPython-dev