[IPython-User] Getting setup on a remote cluster w/ Sun Grid Engine.

MinRK benjaminrk@gmail....
Wed Aug 24 17:07:20 CDT 2011


On Wed, Aug 24, 2011 at 15:05, Dharhas Pothina
<Dharhas.Pothina@twdb.state.tx.us> wrote:
>
> I was able to start the engines and they were submitted to the queue
> properly but I do not have a json file in the corresponding security folder.
> Do I need to do something to generate it.

The JSON file is written by ipcontroller, so it will only show up
after the controller has started.

>
> - dharhas
>
>>>> MinRK <benjaminrk@gmail.com> 8/24/2011 4:44 PM >>>
> On a login node on the cluster:
>
> # create profile with default parallel config files, called sge
> [login] $> ipython profile create sge --parallel
>
> edit IPYTHON_DIR/profile_sge/ipcontroller_config.py, adding the line:
>
> c.HubFactory.ip = '0.0.0.0'
>
> to instruct the controller to listen on all interfaces.
>
> Edit IPYTHON_DIR/profile_sge/ipcluster_config.py, adding the line:
>
> c.IPClusterEngines.engine_launcher_class = 'SGEEngineSetLauncher'
> c.IPClusterStart.controller_launcher_class = 'SGEControllerLauncher'
>
> # optional: specify a queue for all:
> c.SGELauncher.queue = 'short'
> To instruct ipcluster to use SGE to launch the engines and the controller
>
> At this point, you can start 10 engines and a controller with:
>
> [login] $> ipcluster start -n 10 --profile=sge
>
> Now the only file you will need to connect to the cluster will be in:
>
> IPYTHON_DIR/profile_sge/security/ipcontroller_client.json
>
> Just move that file around, and you will be able to connect clients.
> To connect from a laptop, you will probably need to specify a login
> node as the ssh server when you do:
>
> from IPython import parallel
>
> rc = parallel.Client('/path/to/ipcontroller_client.json',
> sshserver='you@login.mycluster.etc')
>
> -MinRK
>
>
> On Wed, Aug 24, 2011 at 13:18, Dharhas Pothina
> <Dharhas.Pothina@twdb.state.tx.us> wrote:
>> Hi All,
>>
>> We have managed to parallelize one of our spatial interpolation scripts
>> very
>> easily with the new ipython parallel. Thanks for developing such a great
>> tool, it was fairly easy to get working. Now we are trying to set things
>> up
>> to run on our internal cluster and I'm having difficulties understanding
>> how
>> to configure things.
>>
>> What I would like to do is have ipython running on a local machine
>> (windows
>> & linux) connect to the cluster, request some nodes through SGE and run
>> the
>> computation. I'm not quite getting what goes where from the documentation.
>>
>> I think I understood the PBS example but I'm still not understanding where
>> I
>> would put the connection information to log into the cluster. I would
>> really
>> appreciate a step by step of what files need to be where and any example
>> config files for an SGE setup.
>>
>> thanks,
>>
>> - dharhas
>>
>>
>>
>>
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>


More information about the IPython-User mailing list