[IPython-User] Using IPython as a Batch Queue
Sat Jan 21 14:33:26 CST 2012
I want to execute many embarrassingly parallel processes. The way I am doing it is the following:
1. Generate scripts
$> ls -lth *.py
-rwx------ 1 petigura staff 181B Jan 20 15:08 grid0000.py*
-rwx------ 1 petigura staff 184B Jan 20 15:08 grid2730.py*
2. Run them in a load balanced way in the following manner.
Convert a script to a python call + log
log = s.split('.')+'.log'
return subprocess.call( 'python %s > %s' % (s,log) ,shell=True )
I've run into a couple of problems:
Periodically, one of my cores drops out. However, when I go back and run it from the shell
$> python script.py
it completes. Is there something that could be hanging the view.map? One of the reasons why I split my jobs up was is if a script fails, subprocess just passes a 1 and presumably view.map would just go on to the next job.
Also, I have a hard time stopping the cluster. Doing
$> ipcluster stop
Doesn't work. What I've been doing is listing all the ipengines and stoping them with the kill command.
Thanks in advance for help/advice!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User