[IPython-User] Can't get ipcluster to run on LAN

[Johann Rohwer]

On Wednesday 08 August 2012 09:42:35 Johann Rohwer wrote:
> On Tuesday 07 August 2012 16:03:52 MinRK wrote:
> > Try removing the old connection files (in `ipython locate profile
> > home_ssh`/security).
> > 
> > There might be an issue when reuse_files=True, that sometimes the
> > old
> > connection files can set the config, so if you ran once with
> > ip=127.0.0.1, then set reuse_files=True, it's possible that
> > 127.0.0.1 is still being used.
> > 
> > Can you verify the contents of the JSON files?

I think I've finally come to the bottom of this. I've experienced 
similar problems in setting up ipcluster at work (12 nodes linux SSH 
cluster with shared home directory). 

1. Downgrading to IPython 0.12.1, everything works fine, also with the 
default configuration, for the SSH Launcher. 

2. It turns out that there is a problem with option parsing. When I 
run "ipcluster start --ip='*'" just with the default profile on 
localhost, a directory '*'  with inside the profile_default/... 
hierarchy is created in the cwd. Similarly when I run "ipcluster start 
--ip=1.2.3.4'' a directory "1.2.3.4" is created in the cwd with the 
profile_default hierarchy inside.

So it seems that this --ip argument is interpreted as --profile_dir or 
similar. The config setting c.HubFactory.ip has a similar effect.

The cluster still starts on localhost as the engines can read the json 
files from the "wrong" directory, but this obviously all fails when the 
files are to be created on remote machines via SSH as a dir like /* or 
/1.2.3.4 cannot be created on a remote machine (no permission). 

---------
I've tried diffing the 0.12.1 code and 0.13 code to try find what's going 
on. In IPython/parallel/apps/launcher.py the class SSHClusterLauncher 
has a method _remote_profie_dir_default(self) (line 653), that does 
some mangling of the profile dir variable. NB spelling mistake in the 
method name. This method is not present in 0.12.1 AFAICT.

But this can't be the whole story as the error also occurs in a 
default setup just on localhost, which does not call the SSH Launcher. 
Clearly it also has to do with option parsing. I am really not at all 
familiar with the IPython code to investigate this further without 
guidance, but will gladly test any fixes. As said, 0.12.1 works "out of 
the box" as intended.

--Johann
E-pos vrywaringsklousule

Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie.

E-mail disclaimer

This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.