[IPython-dev] Using IPython Cluster with SGE -- help needed
Sun Aug 4 17:17:13 CDT 2013
Once the job is submitted, the client kernels are running. You can't use
qsub anymore, it's ipython's job to feed work job in the kernels. I think
it already behaves that way.
Le 4 août 2013 16:29, "Andreas Hilboll" <email@example.com> a écrit :
> Am 04.08.2013 16:20, schrieb Matthieu Brucher:
> > Hi,
> > I guess we may want to start with the ipython documentation on this
> > topic:
> > Cheers,
> > 2013/8/4 Andreas Hilboll <firstname.lastname@example.org>:
> >> Hi,
> >> I would like to use IPython for calculations on our cluster. It's a
> >> total of 11 compute + 1 management nodes (all running Linux), and we're
> >> using SGE's qsub to submit jobs. The $HOME directory is shared via NFS
> >> between all the nodes.
> >> Even after reading the documentation, I'm unsure about how to get things
> >> running. I assume that I'll have to execute ``ìpcluster -n 16`` on all
> >> compute nodes (they have 16 cores each). I'd have the ipython shell
> >> (notebook won't work due to firewall restrictions I cannot change) on
> >> the management node. But how does the management node know about the
> >> kernels which are running on the compute nodes and waiting for a job?
> >> And how can I tell the management node that it shall use qsub to submit
> >> the jobs to the individual kernels?
> >> As I think this is a common use case, I'd be willing to write up a nice
> >> tutorial about the setup, but I fear I need some help from you guys to
> >> get things running ...
> >> Cheers,
> >> -- Andreas.
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPythonemail@example.com
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> Thanks, Matthieu,
> not sure how I missed that :/ Is there a way to "re-use" the ipengines
> on the nodes, like using qsub to "inject" a job into a (idle) running
> ipengine? That way the sometimes longish startup times of the Python
> interpreter on shared network filesystems could be avoided. Something like
> 1. start idling ipengines on all cores
> 2. use qsub to inject jobs into these
> 3. "cleanup" the ipengine after the job is done, making it ready for
> another job.
> -- Andreas.
> IPython-dev mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-dev