[IPython-User] Limitations (?) of ipython SGE support

Brian Granger ellisonbg@gmail....
Fri Jan 21 12:11:28 CST 2011


>> * Our scheduler has much lower lantency and overhead than that of SGE.

> Hmm... In case of our cluster the longer job running time you specify
> the longer you have to wait for it to get started. So when I submit
> ipengines with expected running time 48h (max) I would have to wait
> quite a while before ipcluster will be operational. What is more if I
> try to run any jobs before the ipengines will start running (and I
> don't know when it is going to happen) they will instantly fail
> (because of lack of ipengines). So I cannot start ipcluster; run my
> python code and go for lunch. I have to run ipcluster and wait until
> the ipengines will start running (which can take few our depending on
> the cluster load) and only then I can run my python code.

Yes, this usage case is horrible for IPython currently.  We definitely
need to think further about how to handle these usage cases.  We (the
IPython devs) tend to run things on multicore machines or small
clusters with few users.  But, maybe Min and I can talk more about the
best way to support these types of things.  IPython does have the
ability to talk to job schedulers, so this wouldn't be too crazy.

> What I would prefer is that ipcontroller would qsub every job and use
> some pythonic IPC method to be informed that job has been completed. I
> have not ye looked into how to implemented yet.

We will think about that.



> Best,
> Chris

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo

More information about the IPython-User mailing list