[IPython-user] Problem calling subprocess from TaskClient

[Justin MacCallum]

Hi,

I have a parallel scientific code that runs on clusters. It currently uses my own (badly designed) parallel communication engine and I'm trying to transition to iPython's TaskClient interface. One part of my code uses the subprocess module to wrap a call to a different piece of software. Unfortunately, this seems to be causing problems for me. I'm running on OS X 10.6.2, with python 2.6.4 and ipython 0.10 as supplied by MacPorts. The following code snippet will reproduce the problem I'm having.

-----
#!/usr/bin/env python

from IPython.kernel import client

tc = client.TaskClient()

@tc.parallel()
def remote_test(input):
    import subprocess
    # I'm obviously not wrapping ls, but I have the same problem with the real binary I'm trying to call
    subprocess.check_call('ls')
    return input

work = range(100)

results = remote_test(work)
-----


The output of this program is:

-----
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/clientconnector.py:43: DeprecationWarning: Importing class Tub directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Tub instead
  self.tub = Tub()
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/taskfc.py:79: DeprecationWarning: Importing class Referenceable1 directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Referenceable instead
  class FCTaskControllerFromTaskController(Referenceable):
Traceback (most recent call last):
  File "./test.py", line 15, in <module>
    results = remote_test(work)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/parallelfunction.py", line 104, in call_function
    return self.mapper.map(self.func, *sequences)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/mapper.py", line 230, in map
    task_results = [self.task_controller.get_task_result(tid) for tid in task_ids]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/taskclient.py", line 93, in get_task_result
    taskid, block)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/twistedutil.py", line 72, in blockingCallFromThread
    return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/internet/threads.py", line 114, in blockingCallFromThread
    result.raiseException()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/python/failure.py", line 326, in raiseException
    raise self.type, self.value, self.tb
OSError: [Errno 4] Interrupted system call
-----



The ipcontroller log file looks like:

-----
2010-02-04 12:42:36-0800 [-] Running task 82 on worker 1
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Task completed: 82
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] distributing Tasks
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Running task 83 on worker 1
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Task 83 failed on worker 1
2010-02-04 12:42:36-0800 [-] distributing Tasks
2010-02-04 12:42:36-0800 [-] Running task 84 on worker 0
2010-02-04 12:42:36-0800 [Negotiation,0,192.168.0.194] Task 84 failed on worker 0
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 85 on worker 1
2010-02-04 12:42:37-0800 [Negotiation,2,192.168.0.194] Task 85 failed on worker 1
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 86 on worker 0
2010-02-04 12:42:37-0800 [Negotiation,0,192.168.0.194] Task 86 failed on worker 0
2010-02-04 12:42:38-0800 [-] distributing Tasks
2010-02-04 12:42:38-0800 [-] Running task 87 on worker 1
-----

Notice that a few of the tasks actually complete, while the majority fail with the strange interrupted system call error.

I'd appreciate any help anyone can offer.

Thanks,
Justin

-----
Justin MacCallum
Post Doctoral Fellow
University of California San Francisco