<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear IPython,<br><font class="Apple-style-span" color="#144fae"><br></font>I want to execute many embarrassingly parallel processes. The way I am doing it is the following:<br><font class="Apple-style-span" color="#144fae"><br></font>1. Generate scripts <br><font class="Apple-style-span" color="#144fae"><br></font> $> ls -lth *.py<br> -rwx------ 1 petigura staff 181B Jan 20 15:08 grid0000.py*<br><font class="Apple-style-span" color="#144fae"><br></font> <snip><br><font class="Apple-style-span" color="#144fae"><br></font> -rwx------ 1 petigura staff 184B Jan 20 15:08 grid2730.py*<br><font class="Apple-style-span" color="#144fae"><br></font>2. Run them in a load balanced way in the following manner. <br><font class="Apple-style-span" color="#144fae"><br></font> def srun(s):<br> """<br> Convert a script to a python call + log<br> """<br> log = s.split('.')[0]+'.log'<br> return subprocess.call( 'python %s > %s' % (s,log) ,shell=True )<br><font class="Apple-style-span" color="#144fae"><br></font> view.map(srun,Scripts,block=True)<br><font class="Apple-style-span" color="#144fae"><br></font>I've run into a couple of problems:<br><font class="Apple-style-span" color="#144fae"><br></font>Periodically, one of my cores drops out. However, when I go back and run it from the shell<br><font class="Apple-style-span" color="#144fae"><br></font> $> python script.py<br><font class="Apple-style-span" color="#144fae"><br></font>it completes. Is there something that could be hanging the view.map? One of the reasons why I split my jobs up was is if a script fails, subprocess just passes a 1 and presumably view.map would just go on to the next job.<br><font class="Apple-style-span" color="#144fae"><br></font>Also, I have a hard time stopping the cluster. Doing <br><font class="Apple-style-span" color="#144fae"><br></font> $> ipcluster stop<br><font class="Apple-style-span" color="#144fae"><br></font>Doesn't work. What I've been doing is listing all the ipengines and stoping them with the kill command.<br><font class="Apple-style-span" color="#144fae"><br></font>Thanks in advance for help/advice!<br><font class="Apple-style-span" color="#144fae"><br></font>Erik<br></div></body></html>