<br><br><div class="gmail_quote">On Tue, Jun 12, 2012 at 10:10 AM, Jose Gomez-Dans <span dir="ltr"><<a href="mailto:jgomezdans@gmail.com" target="_blank">jgomezdans@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
So I have ipython 0.11 in our system, and I would like to do some<br>
trivial parallel processing, making use of ssh and the fact that we<br>
have NFS in our home directories (i.e. ~/.config/ipython is visible in<br>
all hosts in the same place).<br>
<br>
So, I followed the instructions here<br>
<<a href="http://mail.scipy.org/pipermail/ipython-user/2011-December/008833.html" target="_blank">http://mail.scipy.org/pipermail/ipython-user/2011-December/008833.html</a>><br>
1.- Create a new test profile using<br>
$ ipython profile create sshtest --parallel<br>
2.- Edit ipcluster_config.py to look like this<br>
<br>
c = get_config()<br>
<br>
# delay 60 seconds between starting the controller, and starting the engines<br>
c.IPClusterStart.delay = 60<br>
<br>
# start engines with SSH<br>
c.IPClusterEngines.engine_launcher = \<br>
'IPython.parallel.apps.launcher.SSHEngineSetLauncher'<br>
<br>
# You only need to use the SSHController launcher if you are *not*<br>
# running ipcluster from the controller machine<br>
c.IPClusterStart.controller_launcher = \<br>
'IPython.parallel.apps.launcher.SSHControllerLauncher'<br>
c.SSHControllerLauncher.hostname = 'my_hostname'<br></blockquote><div><br></div><div>I believe the above config should be controller_launcher_class and engine_launcher_class.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
# this and above should result in<br>
# $> ssh <a href="mailto:you@conlin.lanl.gov">you@conlin.lanl.gov</a> "ipcontroller --ip=128.165...<br>
#<br>
# add `--reuse` to this only *after* everything appears to be working,<br>
#because it can prevent<br>
# certain changes from having an effect<br>
c.SSHControllerLauncher.program_args = [ '--ip=xx.xx.xx.xx',<br>
'--log-to-file', '--log-level=10' ]<br>
<br>
# if you are not using SSH to launch the controller:<br>
c.LocalControllerLauncher.controller_args = [ '--ip=xx.xx.xx.xx',<br>
'--log-to-file', '--log-level=10']<br>
<br>
c.SSHEngineSetLauncher.engines = {<br>
'sun-node01': 2,<br>
'sun-node02': 2, # etc.<br>
}<br>
<br>
3.- Test running the controller on my_hostname<br>
$ ipcontroller --profile=sshtest<br>
4.- Test connecting an engine from one node (say sun-node02 above):<br>
$ ipengine --profile=sshtest<br>
5.- Steps (3) and (4) are succesful (there's a connection, and I can<br>
pass things to the engine, see it's IP address etc)<br>
<br>
Now, I would like to just use ipcluster to launch the controller and<br>
engines. This doesn't work as expected: if I launch<br>
$ ipcluster --profile=sshtest<br>
<br>
then after ~60s, all I get is ipcluster launching 12 engines on the<br>
controller (the controller has 12 cores, so this might be a default).<br>
However, they are all local. I can ssh into the nodes and launch the<br>
engines using ipcluster engine profile=sshtest. However, this launches<br>
4 engines per node (and not 2, these are 4 core machines).<br>
<br>
I really don't understand what's going on here, but my problem appears<br>
similar to the one Jeremy reported. However, I can only use two cores<br>
in the nodes, so ipcluster engine etc doesn't work for me.<br></blockquote><div><br></div><div>I imagine your issue stems from the config typos above. This is behaving exactly as expected if you had simply not specified the engine/controller launcher classes (which you haven't done, since they are given with the wrong name).</div>
<div><br></div><div>When in doubt, always add `--debug`. I expect you will see "starting 12 engines with LocalEngineSetLauncher (this is actually displayed at the default log-level, at least on master).</div><div><br>
</div><div>Every time you report a config question like this, please include the output of the command with `--debug`.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Additionally, how would one go about giving the engines a particular<br>
"nice" value? If I don't sort this stuff out, I think I might become<br>
very unpopular among my colleagues! ;-)<br></blockquote><div><br></div><div>nice is not something ipcluster exposes, you will have to either create your own launchers or use ipengine directly.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks!<br>
Jose<br>
_______________________________________________<br>
IPython-User mailing list<br>
<a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
</blockquote></div><br>