[IPython-User] Getting setup on a remote cluster w/ Sun Grid Engine.

MinRK benjaminrk@gmail....
Wed Aug 24 16:44:42 CDT 2011


On a login node on the cluster:

# create profile with default parallel config files, called sge
[login] $> ipython profile create sge --parallel

edit IPYTHON_DIR/profile_sge/ipcontroller_config.py, adding the line:

c.HubFactory.ip = '0.0.0.0'

to instruct the controller to listen on all interfaces.

Edit IPYTHON_DIR/profile_sge/ipcluster_config.py, adding the line:

c.IPClusterEngines.engine_launcher_class = 'SGEEngineSetLauncher'
c.IPClusterStart.controller_launcher_class = 'SGEControllerLauncher'

# optional: specify a queue for all:
c.SGELauncher.queue = 'short'
To instruct ipcluster to use SGE to launch the engines and the controller

At this point, you can start 10 engines and a controller with:

[login] $> ipcluster start -n 10 --profile=sge

Now the only file you will need to connect to the cluster will be in:

IPYTHON_DIR/profile_sge/security/ipcontroller_client.json

Just move that file around, and you will be able to connect clients.
To connect from a laptop, you will probably need to specify a login
node as the ssh server when you do:

from IPython import parallel

rc = parallel.Client('/path/to/ipcontroller_client.json',
sshserver='you@login.mycluster.etc')

-MinRK


On Wed, Aug 24, 2011 at 13:18, Dharhas Pothina
<Dharhas.Pothina@twdb.state.tx.us> wrote:
> Hi All,
>
> We have managed to parallelize one of our spatial interpolation scripts very
> easily with the new ipython parallel. Thanks for developing such a great
> tool, it was fairly easy to get working. Now we are trying to set things up
> to run on our internal cluster and I'm having difficulties understanding how
> to configure things.
>
> What I would like to do is have ipython running on a local machine (windows
> & linux) connect to the cluster, request some nodes through SGE and run the
> computation. I'm not quite getting what goes where from the documentation.
>
> I think I understood the PBS example but I'm still not understanding where I
> would put the connection information to log into the cluster. I would really
> appreciate a step by step of what files need to be where and any example
> config files for an SGE setup.
>
> thanks,
>
> - dharhas
>
>
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>


More information about the IPython-User mailing list