[IPython-user] [0:execute]: IOError: [Errno 4] Interrupted system call

[mark starnes]

Hi everyone,

I'm performing blocking file read commands on remote engines, one at a time, but regularly get the
error:

********************************************************************************************************

CompositeError                            Traceback (most recent call last)

<ipython console> in <module>()

/fe2.pyc in setup(self, append)
    365             diskpush({'a':self}, picklemode = 0)  # use mode 0 (others fail)
    366          else:
--> 367             diskpush({'a':self})
    368
    369

/reference.pyc in diskpush(a, targets, block, isanobject, picklemode)
   3359       time.sleep(5.0); mprint('Done. ')
   3360       #mec.execute('tmpdata = dpfile.read()', targets = [i], block = True)
-> 3361       mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True)
   3362       mec.execute('dpfile.close()', targets = [i], block = True)
   3363       #mec.barrier(a)

/usr/local/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
    520         targets, block = self._findTargetsAndBlock(targets, block)
    521         result = blockingCallFromThread(self.smultiengine.execute, lines,
--> 522             targets=targets, block=block)
    523         if block:
    524             result = ResultList(result)

/usr/local/lib64/python2.5/site-packages/IPython/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
     67         @raise: any error raised during the callback chain.
     68         """
---> 69         return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
     70
     71 else:

/usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.pyc in blockingCallFromThread(reactor, f, *a, **kw)
     81     result = queue.get()
     82     if isinstance(result, failure.Failure):
---> 83         result.raiseException()
     84     return result
     85

/usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.pyc in raiseException(self)
    317         information if available.
    318         """
--> 319         raise self.type, self.value, self.tb
    320
    321

CompositeError: one or more exceptions from call to method: execute
[0:execute]: IOError: [Errno 4] Interrupted system call

*******************************************************************************************************************

It doesn't always happen; sometimes it will occur after a few hours of execution (ruining overnight runs).
The active parts of the routine falling over are:

=============================================
for i in targets:  # Sequential engine reads.
   mec.execute('dpfile = bz2.BZ2File(tmpfn, "rb")', targets = [i], block = True)
   mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True)
   mec.execute('dpfile.close()', targets = [i], block = True)

=============================================

with targets = mec.get_ids()



Following the error, subsequent requests to the engines get results like:




/<ipython console> in <module>()

/fe2.pyc in setup(self, append)
    365             diskpush({'a':self}, picklemode = 0)  # use mode 0 (others fail)
    366          else:
--> 367             diskpush({'a':self})
    368
    369

/reference.py in diskpush(a, targets, block, isanobject, picklemode)
   3359       #time.sleep(5.0); mprint('Done. ')
   3360       #mec.execute('tmpdata = dpfile.read()', targets = [i], block = True)
-> 3361       mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True)
   3362       mec.execute('dpfile.close()', targets = [i], block = True)
   3363       #mec.barrier(a)

/usr/local/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
    520         targets, block = self._findTargetsAndBlock(targets, block)
    521         result = blockingCallFromThread(self.smultiengine.execute, lines,
--> 522             targets=targets, block=block)
    523         if block:
    524             result = ResultList(result)

/usr/local/lib64/python2.5/site-packages/IPython/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
     67         @raise: any error raised during the callback chain.
     68         """
---> 69         return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
     70
     71 else:

/usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.pyc in blockingCallFromThread(reactor, f, *a, **kw)
     81     result = queue.get()
     82     if isinstance(result, failure.Failure):
---> 83         result.raiseException()
     84     return result
     85

/usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.pyc in raiseException(self)
    317         information if available.
    318         """
--> 319         raise self.type, self.value, self.tb
    320
    321

CompositeError: one or more exceptions from call to method: execute
[0:execute]: AttributeError: 'module' object has no attribute 'cbook'



and then, when I quit IPython, I get,

Closing threads... Done.
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method RemoteReferenceTracker._refLost of <RemoteReferenceTracker(clid=1,url=pbu://127.0.0.1:24879/uwtv36uev6e7emd45xfz77tt75krrduj)>> ignored




The engines need a restart after this error.  Any ideas on how to fix this, would be appreciated.  It's
hobbling my parallel processing attempts!

Thanks in advance,

Mark.






matplotlib version 0.98.3
verbose.level helpful
interactive is False
units is False
platform is linux2
Could not load matplotlib icon: 'module' object has no attribute 'window_set_default_icon_from_file'
backend QtAgg version 0.9.1
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active
Python 2.5.1 (r251:54863, Jan 10 2008, 18:00:49)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.rc1 -- An enhanced Interactive Python.