[IPython-User] Using IPython as a Batch Queue

Erik Petigura epetigura@berkeley....
Fri Jan 20 17:43:02 CST 2012


Dear IPython,

I want to execute many embarrassingly parallel processes.  The way I am doing it is the following:

1. Generate scripts 

   $> ls -lth *.py
   -rwx------  1 petigura  staff   181B Jan 20 15:08 grid0000.py*

                     <snip>

   -rwx------  1 petigura  staff   184B Jan 20 15:08 grid2730.py*

2. Run them in a load balanced way in the following manner.  

   def srun(s):
       """
       Convert a script to a python call + log
       """
       log = s.split('.')[0]+'.log'
       return subprocess.call( 'python %s > %s' % (s,log) ,shell=True )

   view.map(srun,Scripts,block=True)

I've run into a couple of problems:

Periodically, one of my cores drops out.  However, when I go back and run it from the shell

    $> python script.py

it completes.  Is there something that could be hanging the view.map?  One of the reasons why I split my jobs up was is if a script fails, subprocess just passes a 1 and presumably view.map would just go on to the next job.

Also, I have a hard time stopping the cluster.  Doing 

    $> ipcluster stop

Doesn't work.  What I've been doing is listing all the ipengines and stoping them with the kill command.

Thanks in advance for help/advice!

Erik



More information about the IPython-User mailing list