[IPython-User] Running multiple ipclusters on remote cluster w/ Sun Grid Engine.

MinRK benjaminrk@gmail....
Wed Sep 14 14:12:56 CDT 2011

On Wed, Sep 14, 2011 at 11:13, Fernando Perez <fperez.net@gmail.com> wrote:

> Hi Dharhas,
> On Wed, Sep 14, 2011 at 6:59 AM, Dharhas Pothina
> <Dharhas.Pothina@twdb.state.tx.us> wrote:
> > I ended up writing a script that connected to the cluster and made a copy
> of
> > an already created profile with a new unique name, started ipcluster,
> waited
> > till the json file was created and then retrieved the json file for use
> in a
> > local client, runs my script and then cleans up afterwards.
> >
> > This seems to be working fairly well except when the local script exits
> > because of an error. In that case, I need to log in and stop the engines,
> > clean up files etc manually.
> OK.  We probably should remove the assumption of a 1 to 1 mapping
> between profiles and running clusters, but that will require a fair
> bit of reorganization of code that uses that assumption, so I'm glad
> you found a solution for now.

Yes, it's a pretty big deal that the only thing engines and clients need to
know to connect to a cluster is the profile name.  That is lost entirely if
we allow multiple clusters with a single profile, since profile name becomes
ambiguous.  We would then need to add a second layer of specification for
which controller to use within a given profile, e.g.:

ipengine --profile=mysge --controller-id=12345

I think I could add support for exactly this without much code change at
all, though.

Feature Request opened on GitHub:

> Cheers,
> f
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20110914/e7ff515a/attachment.html 

More information about the IPython-User mailing list