[IPython-User] ipcluster in ssh mode -

MinRK benjaminrk@gmail....
Sun Aug 7 18:43:20 CDT 2011


As I mentioned, it was quite straightforward to add tunneling support, at
least for the simplest case:

https://github.com/ipython/ipython/pull/685

:)

-MinRK

On Sun, Aug 7, 2011 at 15:17, MinRK <benjaminrk@gmail.com> wrote:

>
>
> On Sun, Aug 7, 2011 at 14:25, Manuel Jung <mjung@astrophysik.uni-kiel.de>wrote:
>
>> Well no answers yet, but i made some progression.
>>
>> I was not able to work around the error, but i think i understand now, why
>> this does not work.
>>
>> The error appears, because the registration is successfull, but everything
>> else like heartbeat etc. fails. For these operations were no ports
>> forwarded.
>>
>> It is stated here
>>
>>
>> http://ipython.org/ipython-doc/stable/parallel/parallel_securitystandard.html#ssh<http://ipython.org/ipython-doc/stable/parallel/parallel_security.html#ssh>
>>
>> that tunneling for engines (which i tried) is not supported atm. I tried
>> to work around this, but only created a tunnel for the registration socket -
>> not for the other sockets, which are used by the engines. An overview of
>> them is given here:
>>
>>
>> http://ipython.org/ipython-doc/stable/development/parallel_connections.html#all-connections
>>
>> Well i did specify the registration port, but i did not specify ports
>> heartbeats etc. Am i able to do this to get homebrew engine tunneling? I saw
>> a bunch of options which are maybe related in the configuration for the
>> controller, but did'nt quite understud, which ones i had to alter.
>>
>> Maybe someone could point out, why there is no tunneling support for
>> engines there (yet)? Is there any particular reason for this, other than
>> just nobody did it yet?
>>
>
>  Correct, some amount of ssh tunneling will be added to the engine, it just
> hasn't been done.  The reason it's a lower priority than the
> client-controller connections is just that it's more rare that engines can't
> see the controller directly.  It's also slightly less valuable, because
> engines are often run in environments that cannot accept input, so only
> passwordless ssh will work.  The client tunnels allow for input of a
> password (though I doubt that it works in every case).
>
>
> As it stands now, there's no way to tell the engine to ignore the
> connection reply from the controller (which contains all of the
> non-registration connection info), so there are some restrictions on how you
> can trick the engine into connecting to different ports.  Essentially you
> will have to set up all 6 forwarded ports, and the Controller must be
> listening on localhost (can be in addition to localhost, e.g. 0.0.0.0 for
> all interfaces).
>
> Prevent the JSON connector file from disambiguating localhost connections
> to the controller's external IP by specifying loopback, e.g.:
>
> ipcontroller --ip=0.0.0.0 --location=127.0.0.1
>
> That way, engines will always try to connect to localhost, regardless of
> where the Controller actually is running, enabling them to use your tunnels.
>
> First, you must specify (or retrieve from the controller's debug output)
> all of the ports the controller is listening on for engine connections:
>
> in ipcontroller_config.py:
> # port-pairs:
> c.HubFactory.iopub
> c.HubFactory.hb
> c.HubFactory.task
> c.HubFactory.mux
> c.HubFactory.control
>
> Then you can specify the tunnels manually (the local ports *must* be the
> same, for now). That will be the first port of each Queue (iopub, task, mux,
> control) and both hb ports, and the registration port.
>
> So, I was able to get this running with the following commands:
>
> 1. start the controller, listening on all interfaces and forcing loopback
> IP for disambiguation:
>
>  [controller] $> ipcontroller --ip=0.0.0.0 --location=127.0.0.1
> --port=10101 --HubFactory.hb=10102,10112 --HubFactory.control=10203,10103
> --HubFactory.mux=10204,10104 --HubFactory.task=10205,10105
>
> # (with this pattern, 101XY ports are ports visible to the engine, 102XY
> are client-only)
>
> 2. Set up forwarded ports on the engines.
>
> [engine] $> for port in 10101 10102 10112 10103 10104 10105; do ssh $server
> -f -N -L $port:$controller:$port; done
>
> In my case, $server was a third machine that I have ssh access to that has
> access to $controller, where the controller process is running.  If you are
> tunneling directly, then $server would be the controller's IP, and
> $controller would be 127.0.0.1
>
> 3. connect the engine
>
> [engine] $>  ipengine --f=/path/to/ipcontroller-engine.json
>
> # note that if you are on a shared filesystem, just `ipengine` should work.
>
> Implementing support for the easiest case should be quite straightforward,
> and less tedious than this. (Pull requests welcome!).
>
> I hope that helps.
>
> -MinRK
>
>
>> Thanks!
>> Manuel
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20110807/a88a9130/attachment.html 


More information about the IPython-User mailing list