[IPython-User] ipcluster in ssh mode -
Sun Aug 7 18:43:20 CDT 2011
As I mentioned, it was quite straightforward to add tunneling support, at
least for the simplest case:
On Sun, Aug 7, 2011 at 15:17, MinRK <firstname.lastname@example.org> wrote:
> On Sun, Aug 7, 2011 at 14:25, Manuel Jung <email@example.com>wrote:
>> Well no answers yet, but i made some progression.
>> I was not able to work around the error, but i think i understand now, why
>> this does not work.
>> The error appears, because the registration is successfull, but everything
>> else like heartbeat etc. fails. For these operations were no ports
>> It is stated here
>> that tunneling for engines (which i tried) is not supported atm. I tried
>> to work around this, but only created a tunnel for the registration socket -
>> not for the other sockets, which are used by the engines. An overview of
>> them is given here:
>> Well i did specify the registration port, but i did not specify ports
>> heartbeats etc. Am i able to do this to get homebrew engine tunneling? I saw
>> a bunch of options which are maybe related in the configuration for the
>> controller, but did'nt quite understud, which ones i had to alter.
>> Maybe someone could point out, why there is no tunneling support for
>> engines there (yet)? Is there any particular reason for this, other than
>> just nobody did it yet?
> Correct, some amount of ssh tunneling will be added to the engine, it just
> hasn't been done. The reason it's a lower priority than the
> client-controller connections is just that it's more rare that engines can't
> see the controller directly. It's also slightly less valuable, because
> engines are often run in environments that cannot accept input, so only
> passwordless ssh will work. The client tunnels allow for input of a
> password (though I doubt that it works in every case).
> As it stands now, there's no way to tell the engine to ignore the
> connection reply from the controller (which contains all of the
> non-registration connection info), so there are some restrictions on how you
> can trick the engine into connecting to different ports. Essentially you
> will have to set up all 6 forwarded ports, and the Controller must be
> listening on localhost (can be in addition to localhost, e.g. 0.0.0.0 for
> all interfaces).
> Prevent the JSON connector file from disambiguating localhost connections
> to the controller's external IP by specifying loopback, e.g.:
> ipcontroller --ip=0.0.0.0 --location=127.0.0.1
> That way, engines will always try to connect to localhost, regardless of
> where the Controller actually is running, enabling them to use your tunnels.
> First, you must specify (or retrieve from the controller's debug output)
> all of the ports the controller is listening on for engine connections:
> in ipcontroller_config.py:
> # port-pairs:
> Then you can specify the tunnels manually (the local ports *must* be the
> same, for now). That will be the first port of each Queue (iopub, task, mux,
> control) and both hb ports, and the registration port.
> So, I was able to get this running with the following commands:
> 1. start the controller, listening on all interfaces and forcing loopback
> IP for disambiguation:
> [controller] $> ipcontroller --ip=0.0.0.0 --location=127.0.0.1
> --port=10101 --HubFactory.hb=10102,10112 --HubFactory.control=10203,10103
> --HubFactory.mux=10204,10104 --HubFactory.task=10205,10105
> # (with this pattern, 101XY ports are ports visible to the engine, 102XY
> are client-only)
> 2. Set up forwarded ports on the engines.
> [engine] $> for port in 10101 10102 10112 10103 10104 10105; do ssh $server
> -f -N -L $port:$controller:$port; done
> In my case, $server was a third machine that I have ssh access to that has
> access to $controller, where the controller process is running. If you are
> tunneling directly, then $server would be the controller's IP, and
> $controller would be 127.0.0.1
> 3. connect the engine
> [engine] $> ipengine --f=/path/to/ipcontroller-engine.json
> # note that if you are on a shared filesystem, just `ipengine` should work.
> Implementing support for the easiest case should be quite straightforward,
> and less tedious than this. (Pull requests welcome!).
> I hope that helps.
>> IPython-User mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User