[IPython-User] Parallel ipython over ssh+NFS

Jose Gomez-Dans jgomezdans@gmail....
Tue Jun 12 14:00:18 CDT 2012


Hi

Thanks for your reply. There are improvements, but not quite there yet...

On 12 June 2012 19:31, MinRK <benjaminrk@gmail.com> wrote:
> I believe the above config should be controller_launcher_class and
> engine_launcher_class.
> I imagine your issue stems from the config typos above.  This is behaving
> exactly as expected if you had simply not specified the engine/controller
> launcher classes (which you haven't done, since they are given with the
> wrong name).
>
> When in doubt, always add `--debug`.  I expect you will see "starting 12
> engines with LocalEngineSetLauncher (this is actually displayed at the
> default log-level, at least on master).

I don't think ipcluster has a --debug option in 0.11 (it complains it
doesn't understand it). Now, after solving those typos, and running
ipcluster start --profile=ssh test, this comes up on the screen:

[IPClusterStart] Using existing profile dir:
u'/home/ucfajlg/.config/ipython/profile_sshtest'
[IPClusterStart] Starting ipcluster with [daemon=False]
[IPClusterStart] Creating pid file:
/home/ucfajlg/.config/ipython/profile_sshtest/pid/ipcluster.pid
[IPClusterStart] Process 'ssh' started: 16737
[IPClusterStart] tcgetattr: Invalid argument
[IPClusterStart] [IPControllerApp] Config changed:
[IPClusterStart] [IPControllerApp] {'Application': {'log_level': 10},
'HubFactory': {'ip': u'xx.xx.xx.xx'}, 'BaseParallelApplication':
{'log_to_file': True}}
[IPClusterStart] [IPControllerApp] Using existing profile dir:
u'/home/ucfajlg/.config/ipython/profile_default'
[IPClusterStart] [IPControllerApp] Attempting to load config file:
ipython_config.py
[IPClusterStart] [IPControllerApp] Config changed:
[IPClusterStart] [IPControllerApp] {'Application': {'log_level': 10},
'HubFactory': {'ip': u'xx.xx.xx.xx'}, 'TerminalIPythonApp':
{'extensions': ['kernmagic']}, 'ProfileDir': {},
'BaseParallelApplication': {'log_to_file': True}}
[IPClusterStart] [IPControllerApp] Attempting to load config file:
ipcontroller_config.py
[IPClusterStart] Scheduler started [leastload]
[IPClusterStart] Starting 12 engines
[IPClusterStart] Process 'ssh' started: 16780
[IPClusterStart] Starting SSHEngineSetLauncher: ['ssh', '-tt',
u'sun-node08, '/opt/epd-7.1-2-rh5-x86_64/bin/python',
u'/opt/epd-7.1-2-rh5-x86_64/lib/python2.7/site-packages/IPython/parallel/apps/ipengineapp.py',
'--log-to-file', '--log-level=20']
[IPClusterStart] Process 'ssh' started: 16781
[IPClusterStart] Process 'ssh' started: 16782
[...]
[IPClusterStart] Process 'ssh' started: 16807
[IPClusterStart] Process 'engine set' started: [None, None, None,
None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None,
None, None, None]
[IPClusterStart] tcgetattr: Invalid argument
[IPClusterStart] tcgetattr: Invalid argument
[...]
[IPClusterStart] [IPEngineApp] Using existing profile dir:
u'/home/xxx/.config/ipython/profile_default'
[IPClusterStart] [IPEngineApp] Using existing profile dir:
u'/home/xxx/.config/ipython/profile_default'

So, it appears to launch ssh processes galore, which is good. But
what's this about tcgetattr? Anyway, trying to connect works with
rc = Client ()
# OK, I can see the remote engines
but fails if I specify the profile
rc = Client ( profile="sshtest")
TimeoutError: Hub connection request timed out

I can get it to work ignoring the profile option, but I wonder whether
I should be wary of this?

>> Additionally, how would one go about giving the engines a particular
>> "nice" value? If I don't sort this stuff out, I think I might become
>> very unpopular among my colleagues! ;-)
>
>
> nice is not something ipcluster exposes, you will have to either create your
> own launchers or use ipengine directly.

In previous versions, there used to be sshx.sh where you could do this
(also important to set up paths and stuff like that). Is this
documented somewhere?

Thanks for your prompt and helpful answer again!

J


More information about the IPython-User mailing list