[IPython-User] Getting setup on a remote cluster w/ Sun Grid Engine.

Dharhas Pothina Dharhas.Pothina@twdb.state.tx...
Wed Aug 24 17:05:12 CDT 2011

I was able to start the engines and they were submitted to the queue properly but I do not have a json file in the corresponding security folder. Do I need to do something to generate it. 

- dharhas

>>> MinRK <benjaminrk@gmail.com> 8/24/2011 4:44 PM >>>
On a login node on the cluster:

# create profile with default parallel config files, called sge
[login] $> ipython profile create sge --parallel

edit IPYTHON_DIR/profile_sge/ipcontroller_config.py, adding the line:

c.HubFactory.ip = ''

to instruct the controller to listen on all interfaces.

Edit IPYTHON_DIR/profile_sge/ipcluster_config.py, adding the line:

c.IPClusterEngines.engine_launcher_class = 'SGEEngineSetLauncher'
c.IPClusterStart.controller_launcher_class = 'SGEControllerLauncher'

# optional: specify a queue for all:
c.SGELauncher.queue = 'short'
To instruct ipcluster to use SGE to launch the engines and the controller

At this point, you can start 10 engines and a controller with:

[login] $> ipcluster start -n 10 --profile=sge

Now the only file you will need to connect to the cluster will be in:


Just move that file around, and you will be able to connect clients.
To connect from a laptop, you will probably need to specify a login
node as the ssh server when you do:

from IPython import parallel

rc = parallel.Client('/path/to/ipcontroller_client.json',


On Wed, Aug 24, 2011 at 13:18, Dharhas Pothina
<Dharhas.Pothina@twdb.state.tx.us> wrote:
> Hi All,
> We have managed to parallelize one of our spatial interpolation scripts very
> easily with the new ipython parallel. Thanks for developing such a great
> tool, it was fairly easy to get working. Now we are trying to set things up
> to run on our internal cluster and I'm having difficulties understanding how
> to configure things.
> What I would like to do is have ipython running on a local machine (windows
> & linux) connect to the cluster, request some nodes through SGE and run the
> computation. I'm not quite getting what goes where from the documentation.
> I think I understood the PBS example but I'm still not understanding where I
> would put the connection information to log into the cluster. I would really
> appreciate a step by step of what files need to be where and any example
> config files for an SGE setup.
> thanks,
> - dharhas
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
IPython-User mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20110824/a05a5c8b/attachment.html 

More information about the IPython-User mailing list