[Nipy-devel] [nipype] better cluster support proposal
Chris Filo Gorgolewski
Thu Jan 27 10:32:00 CST 2011
On 26 January 2011 21:40, Michael Hanke <email@example.com> wrote:
> On Wed, Jan 26, 2011 at 08:04:35PM +0000, Chris Filo Gorgolewski wrote:
>> I was thinking about our (my?) problems with SGE, and after googling
>> some option I came with this very simple idea:
>> For every node submit a script with "qsub -hold_jid" specifying nodes
>> dependencies. The script will do the following things:
>> 1. Unpickle the interface it is going to run.
>> 2. Collect its inputs (we'll have to somehow pickle its connection info).
>> 3. Check the hash if incorrect run the node.
>> This should scale much better than the current ipython solution,
>> although it changes the architecture slightly.
>> What do you think? Does the cluster you use at Dartmouth support dependencies?
> There are basically two systems. One where you specify job dependencies
> after submission of previous jobs and one where you compose a DAG a
> priori. Both NiPyPE and Condor use the later paradigm.
> I think, it would be more portable to be able to export the entire DAG
> from NiPyPE into some abstract representation. If we had that, we could
> "translate" this DAG into a job array for SGE, into whatever PBS does,
> into Condor's DAGMAN format, etc... -- without having to interface with a
> particular cluster engine deep inside NiPyPE.
Good point. I'm also considering using some IPC method for passing the
inputs insdead of reading the reslts.npz files. Does your cluster
More information about the Nipy-devel