[IPython-User] ipcluster in ssh mode in ipython 0.13

M. Wimmer wimmer.mike@gmail....
Wed Jul 18 09:47:49 CDT 2012


Dear all,

Having freshly installed ipython 0.13 I tried to get an ipython parallel cluster
running via ssh. For this I followed precisely the documentation and did

 ipython profile create --parallel --profile=ssh

and modified, according to the docs, in ipcluster_config.py the lines

 c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher'

 c.SSHEngineSetLauncher.engines = {'xxx049': 1}

Trying to start a cluster then misarebly fails

 ipcluster start --profile=ssh --debug 

and I've added the debug output below. The failed `scp`-commands I can avoid by 
writing in ipcluster_config.py

 c.SSHLauncher.to_send = []
 c.SSHLauncher.to_fetch = []

As you can see from the logfile below, the engines are not started because of
arguments (""--profile ...") being added to the python command. I can change the
engine_cmd to `ipengine` but then I ran into other troubles, so I wanted to
discuss this step by step.

What am I doing wrong? What should be added to the docs?

- Michael

Logfile:
---------------------------------------------------------
2012-07-18 16:40:33,001.001 [IPClusterStart] Config changed:
2012-07-18 16:40:33,001.001 [IPClusterStart] {'Application': {'log_level': 10},
 'BaseIPythonApplication': {'profile': u'ssh'}}
2012-07-18 16:40:33,003.003 [IPClusterStart] Using existing profile dir:
 u'/home/wimmer/.ipython/profile_ssh'
2012-07-18 16:40:33,003.003 [IPClusterStart] Searching path [u'/home/wimmer',
 u'/home/wimmer/.ipython/profile_ssh'] for config files
2012-07-18 16:40:33,003.003 [IPClusterStart] Attempting to load config file:
 ipython_config.py
2012-07-18 16:40:33,005.005 [IPClusterStart] Loaded config file:
 /home/wimmer/.ipython/profile_ssh/ipython_config.py
2012-07-18 16:40:33,005.005 [IPClusterStart] Attempting to load config file:
 ipcluster_config.py
2012-07-18 16:40:33,007.007 [IPClusterStart] Loaded config file:
 /home/wimmer/.ipython/profile_ssh/ipcluster_config.py
2012-07-18 16:40:33,009.009 [IPClusterStart] Config changed:
2012-07-18 16:40:33,009.009 [IPClusterStart] {'Application': {'log_level': 10},
 'BaseIPythonApplication': {'profile': u'ssh'}, 'IPClusterEngines':
 {'engine_launcher_class': 'SSHEngineSetLauncher'}, 'SSHEngineSetLauncher': 
 {'engines': {'xxx049': 2}}, 'ProfileDir': {}}
2012-07-18 16:40:33.012 [IPClusterStart] Starting ipcluster with [daemon=False]
2012-07-18 16:40:33.012 [IPClusterStart] Creating pid file:
 /home/wimmer/.ipython/profile_ssh/pid/ipcluster.pid
2012-07-18 16:40:33.013 [IPClusterStart] Starting Controller with
 LocalControllerLauncher
2012-07-18 16:40:33.013 [IPClusterStart] Starting LocalControllerLauncher:
 ['/usr/bin/python', '-c', 'from IPython.parallel.apps.ipcontrollerapp import
 launch_new_instance; launch_new_instance()', '--profile-dir',
 u'/home/wimmer/.ipython/profile_ssh', '--cluster-id', u'', '--log-to-file',
 '--log-level=20']
2012-07-18 16:40:33.020 [IPClusterStart] Process '/usr/bin/python' started:  
 16130
2012-07-18 16:40:33.747 [IPClusterStart] 2012-07-18 16:40:33,746.746
 [IPControllerApp] Using existing profile dir: u'/home/wimmer/.ipython 
 /profile_ssh'
2012-07-18 16:40:34.012 [IPClusterStart] Starting 2 Engines with
 SSHEngineSetLauncher
2012-07-18 16:40:34.017 [IPClusterStart] waiting for
 /home/wimmer/.ipython/profile_ssh/security/ipcontroller-client.json
2012-07-18 16:40:35.021 [IPClusterStart] sending
 /home/wimmer/.ipython/profile_ssh/security/ipcontroller-client.json to
 xxx049:security/ipcontroller-client.json
 scp: security/ipcontroller-client.json: No such file or directory
2012-07-18 16:40:36.792 [IPClusterStart] sending
 /home/wimmer/.ipython/profile_ssh/security/ipcontroller-engine.json to
 xxx049:security/ipcontroller-engine.json
 scp: security/ipcontroller-engine.json: No such file or directory
2012-07-18 16:40:37.957 [IPClusterStart] Starting SSHEngineLauncher: ['ssh',
 '-tt', u'xxx049', '/usr/bin/python', '-c', 'from
 IPython.parallel.apps.ipengineapp import launch_new_instance;
 launch_new_instance()', '--profile-dir', u'', '--log-to-file', 
 '--log-level=20']
2012-07-18 16:40:37.969 [IPClusterStart] Process 'ssh' started: 16235
2012-07-18 16:40:38.075 [IPClusterStart] Starting SSHEngineLauncher: ['ssh',
 '-tt', u'xxx049', '/usr/bin/python', '-c', 'from
 IPython.parallel.apps.ipengineapp import launch_new_instance;
 launch_new_instance()', '--profile-dir', u'', '--log-to-file', 
 '--log-level=20']
2012-07-18 16:40:38.085 [IPClusterStart] Process 'ssh' started: 16237
2012-07-18 16:40:38.086 [IPClusterStart] Process 'engine set' started: [None,  
 None]
2012-07-18 16:40:38.086 [IPClusterStart] 2012-07-18 16:40:34.055 [scheduler]
 Scheduler started [leastload]
2012-07-18 16:40:39.249 [IPClusterStart] bash: -c: line 0: syntax error near
 unexpected token `--profile-dir'
2012-07-18 16:40:39.249 [IPClusterStart] Connection to xxx049 closed.
2012-07-18 16:40:39.250 [IPClusterStart] bash: -c: line 0: `/usr/bin/python -c
 from IPython.parallel.apps.ipengineapp import launch_new_instance;
 launch_new_instance() --profile-dir  --log-to-file --log-level=20'
2012-07-18 16:40:39.273 [IPClusterStart] Process 'ssh' stopped: {'pid': 16235,
 'exit_code': 1}
2012-07-18 16:40:39.451 [IPClusterStart] bash: -c: line 0: syntax error near
 unexpected token `--profile-dir'
2012-07-18 16:40:39.451 [IPClusterStart] Connection to xxx049 closed.
2012-07-18 16:40:39.452 [IPClusterStart] bash: -c: line 0: `/usr/bin/python -c
 from IPython.parallel.apps.ipengineapp import launch_new_instance;
 launch_new_instance() --profile-dir  --log-to-file --log-level=20'
2012-07-18 16:40:39.455 [IPClusterStart] Process 'ssh' stopped: {'pid': 16237,
 'exit_code': 1}
2012-07-18 16:40:39.455 [IPClusterStart] Process 'engine set' stopped:
 {'xxx049/0': {'pid': 16235, 'exit_code': 1}, 'xxx049/1': {'pid': 16237,
 'exit_code': 1}}
2012-07-18 16:40:39.455 [IPClusterStart] 
            Engines shutdown early, they probably failed to connect.
            
            Check the engine log files for output.
            
            If your controller and engines are not on the same machine, you probably
            have to instruct the controller to listen on an interface other than
localhost.
            
            You can set this by adding "--ip='*'" to your
ControllerLauncher.controller_args.
            
            Be sure to read our security docs before instructing your controller
to listen on
            a public interface.
            
2012-07-18 16:40:39.456 [IPClusterStart] IPython cluster: stopping
2012-07-18 16:40:39.553 [IPClusterStart] Process '/usr/bin/python' stopped:
 {'pid': 16130, 'exit_code': 0}
2012-07-18 16:40:42.458 [IPClusterStart] Removing pid file:
 /home/wimmer/.ipython/profile_ssh/pid/ipcluster.pid




More information about the IPython-User mailing list