[IPython-User] Setting up environment variables for IPCluster over ssh

Tim Head betatim@gmail....
Fri Feb 1 05:07:39 CST 2013

Hello all,

I am trying to start a ssh based ipcluster, controller on my laptop
but engines on a remote machine. The tricky bit is that on the remote
machine I need to run several scripts to setup environment variables
so that the right version of python is available and then change into
a virtualenv.

My idea was to create a bash script on the remote machine called
run_ipengine which will first source and setup etc everything and then
run the ipengine command. I have managed to edit the
ipcluser_config.py so that in the debug output of:

$ ipcluster start --profile=lxpcluster --debug

I see the following:

2013-02-01 11:54:12.426 [IPClusterStart] Starting SSHEngineLauncher:
['ssh', '-tt', u'lxplus430.cern.ch', "'echo --profile-dir
.ipython/profile_lxpcluster --log-to-file --log-level=20 >
.ipengine_args; bash run_ipengine'"]

which is indeed what I would like to have executed. However running my
ipcluster start command the next thing I see in the output is the

2013-02-01 11:54:12.534 [IPClusterStart] Starting SSHEngineLauncher:
['ssh', '-tt', u'lxplus430.cern.ch', "'echo --profile-dir
.ipython/profile_lxpcluster --log-to-file --log-level=20 >
.ipengine_args; bash run_ipengine'"]
2013-02-01 11:54:12.538 [IPClusterStart] Process 'ssh' started: 32553
2013-02-01 11:54:12.538 [IPClusterStart] Process 'engine set' started:
[None, None]
2013-02-01 11:54:12.791 [IPClusterStart] ControlSocket
/var/tmp/master-thead@lxplus430.cern.ch-22 already exists, disabling
2013-02-01 11:54:12.896 [IPClusterStart] bash: echo --profile-dir
.ipython/profile_lxpcluster --log-to-file --log-level=20 >
.ipengine_args; bash run_ipengine: No such file or directory
2013-02-01 11:54:12.897 [IPClusterStart] Connection to lxplus430.cern.ch closed.
2013-02-01 11:54:12.897 [IPClusterStart] Process 'ssh' stopped:
{'pid': 32552, 'exit_code': 127}

To check that run_ipengine exists and does what I think it should I
took this and ran " ".join(that_ssh_cmd) and just ran it in a shell it
does indeed start an ipengine on the remote machine (a log file
appears which tells me it can't find the controller to connect to
which makes sense, I think). Next I tried an even simpler thing and
used "echo woof" as engine_cmd but I get the same error:

2013-02-01 11:51:36.051 [IPClusterStart] Starting SSHEngineLauncher:
['ssh', '-tt', u'lxplus430.cern.ch', "'echo woof'"]
2013-02-01 11:51:36.055 [IPClusterStart] Process 'ssh' started: 32507
2013-02-01 11:51:36.055 [IPClusterStart] Process 'engine set' started:
[None, None]
2013-02-01 11:51:36.321 [IPClusterStart] ControlSocket
/var/tmp/master-thead@lxplus430.cern.ch-22 already exists, disabling
2013-02-01 11:51:36.766 [IPClusterStart] /usr/bin/xauth:  error in
locking authority file /afs/cern.ch/user/t/thead/.Xauthority
2013-02-01 11:51:36.768 [IPClusterStart] bash: echo woof: command not found

Now I am just confused :(

This is with ipython 0.13, python 2.7.3 on my laptop where I am
running ipcluster and python 2.6.5 (after setting up the right env
variables) on the remote machine.

Maybe someone else has succeeded in running a script before ipengine
gets executed and has some wisdom to pass on. For example I would love
to know a better way of passing the cluster and engine args other than
dumping them in a file and reading them back in run_ipengine.
Currently I do this because one of the scripts I need to source to
setup the environment reads $@ and fails when it sees --profile-dir.



More information about the IPython-User mailing list