[IPython-User] notebook and parallel work on Windows7 and Linux
Wed Jan 30 06:43:39 CST 2013
I am trying to run an IPython notebook and (with it) use a Linux-cluster
for long-running parallel work.
My PC runs Windows7, and the network connecting it with the cluster
tends to shut down ssh-connections from time to time. I've tried running
the notebook on the cluster (with a ssh-tunnel bringing it to a browser
on my PC) and on my PC (with a ipcontroller running on the cluster,
connected via the built-in ssh).
Both setups have problems:
If I have the notebook running on the cluster ssh-tunneling to a browser
on my PC I don't know how to "reconnect" the browser to the "old
session" when the tunnel breaks down.
If the notebook is running on the PC ssh-ing with the ipcontroller on
the cluster, it seems they silently "loose contact" during long tasks.
Then I can see the cluster finish its tasks (all 80 x 16 hours!), but
the notebook remains "busy" and doesn't receive anything. - No idea how
I could retrieve the results from the ipcontroller, the engines, or
Is it bad to have tasks that run for so long?
Or is there a standard way to distribute notebook and cluster on an
"unstable network" without these problems? - Surely it doesn't make
sense to run a browser on the cluster?!
A smaller issue I noticed in the second configuration is that while
IPython's paramiko can work nicely with PuTTY (rather: Pageant) to find
private ssh keys, it doesn't recognize PuTTY's "known hosts" or remember
its own. (Apparently PuTTY stores them in the registry, but paramiko
would like them saved to a file by the application?) So I keep getting
warnings about an unknown host (first time in the browser-notebook,
later in its cmd-output) although it should be well-known by now.
More information about the IPython-User