[IPython-User] Is iPython useful for this scenario?

[Brian Granger]

Florian,

On Sat, Feb 4, 2012 at 2:03 PM, Florian Lindner <mailinglists@xgm.de> wrote:
> Hello,
>
> I'm currently working on a control-/queue-management software for a
> CFD simulation system. If consists of three parts:
>
> - A client communicates with the CFD system. It can be long running.
> The client can also be run standalone.
>
> - A server which does the queue management and starts up the clients.
> It is non-interactive
>
> - A server interface which the user uses to talk to the server, e.g.
> to enqueue new jobs.

I am following your design here, but the naming of things is a bit
backwards from IPython.  Here is our terminology:

* Engine = runs on a compute node and does the actual computation.
This is where the CFD sim would run.
* Controller = Schedules tasks to engines using lightweight, low
latency scheduler.
* Cluster = Starts Engines/Controller using batch system.
* Client = Frontend process that the users uses to talk to the above.

> Currently they are communicating via XMLRPC (from python stdlib):
>
> client <---- server <---- server interface.

This architecture is partially reinventing everything in
IPython.parallel.  I would just use IPython.parallel and take
advantage of everything we have there.  It is extremely powerful and
will out perform XMLRPC by a long shot.

> A this time the system works only localhost and with one client.

IPython supports a wide range of cluster configurations (PBS, Torque,
mpiexec, SSH, etc.) and multiple engines and clients.

> Before continue to extend it I wonder if iPython could be useful for
> network communication and process management. I browsed through the
> docs but I'm not entirely sure if I got the ideas of iPython right.
>
> The user should not get in contact with iPython. The software is not
> doing and probably will never do any numerical demanding calculations
> itself.
>
> Is iPython useful in the scenario?

It would be extremely useful.  I would check out our cluster docs here:

http://ipython.org/ipython-doc/dev/parallel/index.html

The notebook would also be useful as well:

http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html

Cheers,

Brian

> Thanks!
>
> Florian
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com