[IPython-User] experience using ipython elastically

Jon Olav Vik jonovik@gmail....
Sat Oct 6 08:38:22 CDT 2012

Caius Howcroft <caius.howcroft <at> gmail.com> writes:

> We have been using ipython as the framework for an analysis engine
> based in ec2. Because of the nature of the work we do the number of
> nodes required at any particular stage is highly variable.  Currently,
> we just allow users to start as many nodes as needed and then just
> start ipython cluster with ipcluster (if anyone is interested in how
> we got ipython to "discover" the nodes I can write that up). However
> now I would like to programmatically expand and shrink the cluster as
> needed and I was wondering if people had any experience with this?

Yes. An IPython.parallel.Client will patiently wait for engines to become 
available, retry if engines disappear, and update the list of ids. I have 
primarily been using load_balanced_view(retries=..., ordered=False) to tolerate 
engines disappearing in the middle of my work. Your case of removing engines 
only when they are not busy sounds simpler.

> My current plan is (roughly) this:
> * start up a ipcontroller on the head node
> * allow our framework to start slave nodes (using boto) as needed,
> using user-data/cloud-init  to launch the ipengines.
> * allow our framework to kill off nodes when demand shrinks again

See this outline:


and also this:


More information about the IPython-User mailing list