[IPython-User] Using IPython as a Batch Queue

[Erik Petigura]

Dear IPython,

I want to execute many embarrassingly parallel processes.  The way I am doing it is the following:

1. Generate scripts 

  $> ls -lth *.py
  -rwx------  1 petigura  staff   181B Jan 20 15:08 grid0000.py*

                    <snip>

  -rwx------  1 petigura  staff   184B Jan 20 15:08 grid2730.py*

2. Run them in a load balanced way in the following manner.  

  def srun(s):
      """
      Convert a script to a python call + log
      """
      log = s.split('.')[0]+'.log'
      return subprocess.call( 'python %s > %s' % (s,log) ,shell=True )

  view.map(srun,Scripts,block=True)

I've run into a couple of problems:

Periodically, one of my cores drops out.  However, when I go back and run it from the shell

   $> python script.py

it completes.  Is there something that could be hanging the view.map?  One of the reasons why I split my jobs up was is if a script fails, subprocess just passes a 1 and presumably view.map would just go on to the next job.

Also, I have a hard time stopping the cluster.  Doing 

   $> ipcluster stop

Doesn't work.  What I've been doing is listing all the ipengines and stoping them with the kill command.

Thanks in advance for help/advice!

Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120121/449b3e83/attachment.html