<br><br><div class="gmail_quote">On Tue, Jul 27, 2010 at 10:56 AM, kcsmith <span dir="ltr">&lt;<a href="mailto:kcsmith@raytheon.com">kcsmith@raytheon.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Brian:<br>
<br>
Yes, I have a shared .ipython directory that all compute nodes see.  Only<br>
localhost[127.0.0.1] was listed.<br>
<br></blockquote><div><br></div><div>This is quite odd.  I am not sure what is going on with this.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

I&#39;m not sure how to use the native SGE support.  I&#39;m running version 0.10 of<br>
IPython.<br>
<br></blockquote><div><br></div><div>Just install this branch of IPython:</div><div><br></div><div><a href="http://github.com/jtriley/ipython/tree/0.10.1-sge">http://github.com/jtriley/ipython/tree/0.10.1-sge</a></div><div>
<br></div><div>And then, the SGE stuff is documented here:</div><div><br></div><div><a href="http://github.com/jtriley/ipython/blob/e4e96f6ddc468a09cfb2d910d51345feb11e05bf/docs/source/parallel/parallel_process.txt">http://github.com/jtriley/ipython/blob/e4e96f6ddc468a09cfb2d910d51345feb11e05bf/docs/source/parallel/parallel_process.txt</a></div>
<div><br></div><div>It should be as simple as:</div><div><br></div>ipcluster sge -n 12 -q hpcqueue</div><div class="gmail_quote"><br></div><div class="gmail_quote">Let us know if you run into problems.  I am CC&#39;ing Justin Riley on this email, who has done the SGE support recently.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">Cheers,</div><div class="gmail_quote"><br></div><div class="gmail_quote">Brian</div><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

<br>
Thanks<br>
<div><div></div><div class="h5"><br>
<br>
Brian Granger-3 wrote:<br>
&gt;<br>
&gt; On Tue, Jul 27, 2010 at 10:06 AM, kcsmith &lt;<a href="mailto:kcsmith@raytheon.com">kcsmith@raytheon.com</a>&gt; wrote:<br>
&gt;<br>
&gt;&gt;<br>
&gt;&gt; I got it to work by NOT using ipcluster<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt; Yes, you can definitely use ipcontroller and ipengine themselves.<br>
&gt;<br>
&gt;<br>
&gt;&gt; For those who care, here&#39;s the script I submit via qsub to the Sun Gird<br>
&gt;&gt; Engine on an 80 core, 10 compute node Linux Rocks cluster<br>
&gt;&gt; -----------------------------------------------------------<br>
&gt;&gt; #!/bin/bash<br>
&gt;&gt; #$ -cwd<br>
&gt;&gt; #$ -pe Common 40<br>
&gt;&gt; #$ -j y<br>
&gt;&gt; #$ -S /bin/bash<br>
&gt;&gt; echo &quot;Starting...&quot;<br>
&gt;&gt; echo $HOSTNAME<br>
&gt;&gt; echo $NSLOTS<br>
&gt;&gt; # Note:  Some of the following may not be needed<br>
&gt;&gt; export TMP=/tmp<br>
&gt;&gt; export TMPDIR=/tmp<br>
&gt;&gt; export MPI_DIR=/opt/openmpi/<br>
&gt;&gt; PATH=$PATH:/opt/openmpi/bin:/share/apps/bin:/share/apps/lib<br>
&gt;&gt; export PATH<br>
&gt;&gt; # Note: Sun Grid Engine will pick a compute node to run this on (i.e. NOT<br>
&gt;&gt; the head node under Rocks)<br>
&gt;&gt; ipcontroller -r --client-location=$HOSTNAME --engine-location=$HOSTNAME<br>
&gt;&gt; --client-port=10100 --engine-port=10101 -l=ipcontroller.log &amp;<br>
&gt;&gt; sleep 3<br>
&gt;&gt; echo &quot;starting ipengines...&quot;<br>
&gt;&gt; mpiexec -n $NSLOTS ipengine --mpi=mpi4py<br>
&gt;&gt; wait<br>
&gt;&gt; -------------------------------------------------------------<br>
&gt;&gt;<br>
&gt;&gt; I was mislead by the ipcluster documentation which appears to imply that<br>
&gt;&gt; ipcluster mpiexec -n $NSLOTS --mpi=mpi4py<br>
&gt;&gt; would work when ipengines and client run on different servers.<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt; OK, this is probably a bug.  But please try the native SGE support that we<br>
&gt; have now.<br>
&gt;<br>
&gt; If you see the following error message:<br>
&gt;&gt;<br>
&gt;&gt; Failure: twisted.internet.error.ConnectionRefusedError: Connection was<br>
&gt;&gt; refused by other side: 111: Connection refused.<br>
&gt;&gt;<br>
&gt;&gt; Check your furl files<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt; Yes.<br>
&gt;<br>
&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; kcsmith wrote:<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I&#39;m trying to run ipcluster under the sun grid engine on a 10 node<br>
&gt;&gt; cluster<br>
&gt;&gt; &gt; and I encountered the following error.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Only those ipengines which reside on the same node as ipcontroller<br>
&gt;&gt; &gt; connect.   The rest get CONNECTION REFUSED[111] errors.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I traced this problem down to the furl files that ipcontroller creates.<br>
&gt;&gt; &gt; They only have the local host ip address listed.<br>
&gt;&gt; &gt; pb://<br>
&gt;&gt; <a href="http://d2vqoq6l7tmjtdjl4gi2ctwlwbxzzdc2@127.0.0.1:56104/ei4yhcb5qqa3pyyoi32j3guqfkzqtd5q" target="_blank">d2vqoq6l7tmjtdjl4gi2ctwlwbxzzdc2@127.0.0.1:56104/ei4yhcb5qqa3pyyoi32j3guqfkzqtd5q</a><br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; If I manually add the actual ipcontroller node&#39;s ip address to the furl<br>
&gt;&gt; &gt; then everything works, ipengines connect and the client connects.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; i.e.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; pb://<br>
&gt;&gt; <a href="http://d2vqoq6l7tmjtdjl4gi2ctwlwbxzzdc2@10.0.255.234:56104/ei4yhcb5qqa3pyyoi32j3guqfkzqtd5q" target="_blank">d2vqoq6l7tmjtdjl4gi2ctwlwbxzzdc2@10.0.255.234:56104/ei4yhcb5qqa3pyyoi32j3guqfkzqtd5q</a><br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; When ipcontroller is started on 10.0.255.234<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Is there some system setting or environment variable which can be set<br>
&gt;&gt; to<br>
&gt;&gt; &gt; force foolscap to include the ipcontroller node ip address?  Or is<br>
&gt;&gt; there<br>
&gt;&gt; &gt; something else wrong??<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Thanks,<br>
&gt;&gt; &gt; Keith<br>
&gt;&gt; &gt;<br>
&gt;&gt;<br>
&gt;&gt; --<br>
&gt;&gt; View this message in context:<br>
&gt;&gt; <a href="http://old.nabble.com/furls-only-have-localhost-as-ipcontroller-location-tp29271660p29278568.html" target="_blank">http://old.nabble.com/furls-only-have-localhost-as-ipcontroller-location-tp29271660p29278568.html</a><br>

&gt;&gt; Sent from the IPython - User mailing list archive at Nabble.com.<br>
&gt;&gt;<br>
&gt;&gt; _______________________________________________<br>
&gt;&gt; IPython-User mailing list<br>
&gt;&gt; <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
&gt;&gt; <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
&gt;&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; --<br>
&gt; Brian E. Granger, Ph.D.<br>
&gt; Assistant Professor of Physics<br>
&gt; Cal Poly State University, San Luis Obispo<br>
&gt; <a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a><br>
&gt; <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; IPython-User mailing list<br>
&gt; <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
&gt; <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
&gt;<br>
&gt;<br>
<br>
--<br>
</div></div>View this message in context: <a href="http://old.nabble.com/furls-only-have-localhost-as-ipcontroller-location-tp29271660p29279048.html" target="_blank">http://old.nabble.com/furls-only-have-localhost-as-ipcontroller-location-tp29271660p29279048.html</a><br>

<div><div></div><div class="h5">Sent from the IPython - User mailing list archive at Nabble.com.<br>
<br>
_______________________________________________<br>
IPython-User mailing list<br>
<a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Brian E. Granger, Ph.D.<br>Assistant Professor of Physics<br>Cal Poly State University, San Luis Obispo<br><a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>