[IPython-User] ipython parallel: results never returned

Moritz Emanuel Beber moritz.beber@gmail....
Mon Apr 29 04:24:40 CDT 2013


Hi,

On 04/29/2013 09:33 AM, John Reid wrote:
> Hi,
>
> I have a problem where my client never seems to get all the results it
> has submitted. I submit about 65000 tasks with
>
> rc = parallel.Client()
> lview = rc.load_balanced_view() # default load-balanced view
> lview.block = False
> async_results = [lview.apply(method.run_job, job, options) for job in jobs]
Have you tried the map functions instead? Just a quick thought.
>
> then I wait for the results with:
>
> for job, ar in zip(jobs, async_results):
>      try:
>          assert 1 == len(ar.msg_ids)
>          duration, size = ar.get()
>      except Exception, e:
>          logging.error("Asynchronous error for job %s", job.output_dir)
>          logging.error("%s: %s", type(e), e)
>          logging.error(e.traceback)
>
> but this never completes even though I know the tasks have completed on
> the engines. Am I doing something wrong or is there a problem with
> ipython? None of the jobs raise exceptions.
>
> When I interrupt the client I get this traceback:
>
> ---------------------------------------------------------------------------
> KeyboardInterrupt                         Traceback (most recent call last)
> /home/john/src/ipython/IPython/utils/py3compat.pyc in execfile(fname,
> *where)
>      181             else:
>      182                 filename = fname
> --> 183             __builtin__.execfile(filename, *where)
>
> /home/john/Analysis/PWM-scanning-benchmarks/python/scripts/run_method.py
> in <module>()
>      163     try:
>      164         assert 1 == len(ar.msg_ids)
> --> 165         duration, size = ar.get()
>      166         durations.append(duration)
>      167         sizes.append(size)
>
> /home/john/src/ipython/IPython/parallel/client/asyncresult.pyc in
> get(self, timeout)
>      118         """
>      119         if not self.ready():
> --> 120             self.wait(timeout)
>      121
>      122         if self._ready:
>
> /home/john/src/ipython/IPython/parallel/client/asyncresult.pyc in
> wait(self, timeout)
>      149             self._wait_for_outputs(timeout)
>      150             return
> --> 151         self._ready = self._client.wait(self.msg_ids, timeout)
>      152         if self._ready:
>      153             try:
>
> /home/john/src/ipython/IPython/parallel/client/client.pyc in wait(self,
> jobs, timeout)
>     1056             if timeout >= 0 and ( time.time()-tic ) > timeout:
>     1057                 break
> -> 1058             time.sleep(1e-3)
>     1059             self.spin()
>     1060         return len(theids.intersection(self.outstanding)) == 0
>
> KeyboardInterrupt:
>
> I'm running '1.0.dev' but I've seen similar behaviour with earlier version.
>
> Thanks for any help,
> John.
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user



More information about the IPython-User mailing list