[IPython-User] Limitations (?) of ipython SGE support
Chris Filo Gorgolewski
Wed Jan 19 14:58:04 CST 2011
On 18 January 2011 05:33, Brian Granger <firstname.lastname@example.org> wrote:
>> Thanks for your reply. It seems that interactive use is your main goal.
>> However, wouldn't out be better to submit every task separately using qsub
>> (instead of submitting ipengines)?
> There are a couple of reasons for doing this (just one long task):
> * Our scheduler has much lower lantency and overhead than that of SGE.
Hmm... In case of our cluster the longer job running time you specify
the longer you have to wait for it to get started. So when I submit
ipengines with expected running time 48h (max) I would have to wait
quite a while before ipcluster will be operational. What is more if I
try to run any jobs before the ipengines will start running (and I
don't know when it is going to happen) they will instantly fail
(because of lack of ipengines). So I cannot start ipcluster; run my
python code and go for lunch. I have to run ipcluster and wait until
the ipengines will start running (which can take few our depending on
the cluster load) and only then I can run my python code.
What I would prefer is that ipcontroller would qsub every job and use
some pythonic IPC method to be informed that job has been completed. I
have not ye looked into how to implemented yet.
More information about the IPython-User