[IPython-User] IPython Parallel on SGE (or PBS) and on-demand scheduling

William Furnass will@thearete.co...
Sat Mar 9 07:20:01 CST 2013


> Date: Tue, 19 Feb 2013 16:57:13 +0100
> From: Martijn Vermaat <m.vermaat.hg@lumc.nl>
> Subject: [IPython-User] IPython Parallel on SGE (or PBS) and on-demand
>         scheduling>
...
> This could be implemented, I think, by having SGEEngineSetLauncher
> schedule individual engines as separate SGE jobs, instead of scheduling
> one big fixed job array. Dynamic scaling of the engine pool would
> require some additional complexity.
>
> Without having much experience with IPython Parallel, I have the feeling
> this doesn't really fit its parellalism model (the pool of engines is
> fixed?). But I hope I'm wrong.

A little while ago I looked at using SGE for some embarrassingly
parallel problems and came to the same conclusion i.e. that I don't
want to have to wait until say 30 cores can be 'booked' simultaneously
before my job can be run.  Should similar computational needs arise
again I'll probably try packaging my problem so that the input
parameters that vary between execution 'strands' are read in from
command-line arguments, thus allowing me to submit a SGE Job Array
[1].

[1] http://www.sheffield.ac.uk/wrgrid/using/runapps/examples2

Will


More information about the IPython-User mailing list