[IPython-dev] Some ideas for more hooks

Frédéric Mantegazza mantegazza at ill.fr
Wed Mar 16 09:56:16 CST 2005

Hello Fernando,

Here is a little paper to explain the (unusual) way we are using ipython in 
our application (PyMAD). This is a summary of our needs of internal 
interaction with ipython to make something very powerfull.

I hope it will help you to re-design ipython with user customizable hooks.

PS : PyMAD is a software we are developping to control spectrometers, big 
neutrons instruments.

-------------- next part --------------
This document describes the interactions needed by PyMAD with ipython.

1) Introduction

The philosophy of PyMAD is to give to the user a high-level set of tools to
drive a spectrometer. The idea is to directly make some internal python
objects available to the user, and let him combine them.

So, The final user just calls some methods of a few high-level objects to drive
the spectrometer. This as the advantage to make all the python scripting stuff
available, to build some new high level tools.

As we also need a system which can be used from several places, we use a
client/server framework, with Pyro. Pyro is a distributed objects server. It
just make some remote python objects available to a client as if they where

2) User interaction

To avoid the need for the final user to write python scripts and run them to do
something, we need a simple working environment which gives the possibility to
interactively use the server objects. That's where ipython solves lots of

ipython is an enhanced python shell. It let the user runs python code, but has
many powerfull features :

 - history, even between sessions
 - colorized and customizable traceback
 - code completion
 - magic commands
 - much more...
ipython is on the client-side of PyMAD. In fact, there is a special client which
connects to the remote objects of PyMAD, launch an embbeded ipython instance,
and make the remote objects available to the user, in the global namespace of
ipython. This way, the user can drive the spectrometer through the methods of
these objects, or can build complex scripts to do complex actions.

PyMAD also use the magic commands to make a simple MAD-like command interpreter.
The magic functions use TPG (Toy Parser Generator), a easy-to-use parser based
on regexp. These MAD-like commands are for users which don't know about python,
but also to make shortcuts, to avoid the need to write several lines of normal
python code to do some complex but repetitive tasks.

One important point is that PyMAD can understand both syntax, which can be
combined. Most of the time, MAD-like commands will be used, but python code can
be more powerfull to do expert measures (with automatic feedback interaction
according to the results), or to prototype a new complex command.

3) ipython needs

In order to give users all these powerfull features, PyMAD needs to interacts
very closely with ipython. In the actual state of ipython we patch some internal
classes, by rebinding methods to custom ones. This is not very clean, and can
lead to problems if internal structures of new ipyton releases change.

So, here is the main PyMAD interactions needed :

  Catch custom (PyMAD) exceptions (now done with rebinding
IPython.iplib.InteractiveShell.runcode() method), with the possibility to
get the inital text code involved in the exception. For the moment, in the
runcode() method, we only get the code object, from which it is impossible to
retreive the text.

  Add some new matchers for completion. As PyMAD use remote
objects, completion only shows the client Pyro proxy. So we added a new
matcher by adding a IPython.iplib.MagicCompleter.proxy_matches() method,
and insert this matcher in ipshell.IP.Completer.matchers list. The new
matcher get the object (from the text param), call a special method on this
object which returns all available attributes (in fact, only these we want
to show to the user). But it does not work anymore in 0.6.12 :o(

  In the same way as matchers, get the docstring from the
remote object instead of the client one when using 'object?' syntaxe.(don't now
how to do this, but it could be done on the same idea: calling a special
method on the object, method returning the doc of our remote object).

  New exception handler.

  Prevent objects from beeing deleted from global namespace.

  Prompt modification at run-time (already solved. I have to do some tests)
  Access to the command-line interpreter, to have ipython interprets code has if
  it was entered through keyboard (ie make difference between magic commands and
  normal python code).

More information about the IPython-dev mailing list