[SciPy-dev] pynotes

Arnd Baecker arnd.baecker at web.de
Sun Nov 7 06:59:05 CST 2004


Hi,

in the ASP thread I suggested to have some scheme
to add notes to scipy.
To test this out I have set up a proof-of-concept:
After importing pynotes (from pynotes import *) one can
use
  addnote(obj)
to invoke an editor (xemacs at this point)
in wich one can add comments to the module/function <obj>.

This information will be displayed when calling
  notehelp(obj).

For each help text a nested structure of folders is created
in ~/.pynotes.

With
  shownotes()
all available notes are listed.
Searching for a string in the notes is possible with
  findnotes(searchstring)

All this has not been tested extensively, but thoses
cases I tried worked fine. (I would be very surprised
if this would be working under windows at the moment,
even though only small changes to construct_path()
would be needed, I think).

Problems at the moment are related to
the construction of paths for certain objects
(the path construction is really the heart of this approach)
- builtin's, like for example math.cos.
  (This can be overcome because math.cos.__module__
  exists, so we can get the path of the math module
  and add a subdir for `cos`.)
- I have no idea how to construct a path for `ufuncs`.
  For example
     import Numeric
     Numeric?
     Numeric.cos.<TAB>

     inspect.getsource(Numeric.cos)     # does not work
     inspect.getfile(Numeric.cos)       # does not work
     inspect.getmodulename(Numeric.cos) # does not work
  (the same happens for example for scipy.special.j0)

  Technically this seems the biggest show-stopper to me,
  so any suggestions/ideas are greatly appreciated here!

In the longer run the idea is to integrate this into IPython
(which will solve quite a few problems) - Fernando
was very positive of a sneak-preview which I sent to him yesterday.

Before doing so I would like to hear your opinions,
feature wishes etc.
Particularly important to me is that this frame-work
can be used in combination with Travis' Python LiveDocs.

More generally, the approach to add user-comments
to many python modules/functions is of course
independent of scipy and the code was written with this is mind.

Looking forward to feedback,

Arnd


#### pynotes.py #####################################
"""
pynotes..py - add private notes to python modules and functions

Arnd Baecker (2004)

Warning: this is a proof-of-concept...


Approach
========

Using the command
  addnote(obj)
an editor is invoked in wich one can add comments to the module/function

This information will be displayed when calling
notehelp.
For each help text a nested structure of folders is created
in ~/.pynotes.

With
  shownotes()
all available notes are listed.
Searching for a string in the notes is possible with
  findnotes(searchstring)

Long Term perspective:
All this is to be integrated seamlessly into IPythons help.

Usage
=====

Usage example 1:

In [1]: from pynotes import *
In [2]: import Numeric
In [3]: addnote(Numeric)
In [4]: notehelp(Numeric)
Numeric module defining a multi-dimensional array and useful procedures
for
   Numerical computation.

Functions
[...]
====== additional comment =====
This module is absolutely essential for any numerical
work in python (see also numarray).
For simplest use a
   from Numeric import *
is not completely discouraged ;-)
==============================



Implementation notes
====================

- For a given object its path is determined (hopefully unique)
- when adding a note the corresponding path segment is created under
  ~/.pynote
- when help is requested for an object it is checked
  if an additional comment is available, and displayed.


TODO
====

- allow change of editor
- construct_path has to be improved.
  For example I don't know how to deal with ufuncs ...
- add routine to the Ipython help to see the added documentation

- no formatting done etc.
  (not really necessary, as I think it would be best
  to plug the notehelp stuff into IPython's normal
  help via _inspect in Magic.py

  (called from magic_pinfo, which in turn is called from
   handle_help in iplib.py)


Some problems - examples
========================

- adding a note to scipy.xplt leads to
    ~/.pynote/site-packages/scipy/xplt/__init__.txt
  (I am not sure if this filename  __init__.txt  really problematic,
   at least it does work ;-)
- We can't deal with types 'builtin_function_or_method'
  Example:
    In [1]: import math
    In [2]: type(math.cos)
    Out[2]: <type 'builtin_function_or_method'>

  Reason: inspect.getsourcefile(math.cos) fails
  Possible solution:
      math.cos.__module__ exists (and is `math`)
  then we only have to use
      eval(math.cos.__module__).__file__
  (But this does not work in the name_space of the routine getting
  the path..., but it will in IPython ...)

- We can't deal with types 'ufunc'
  Example:
    In [1]: import Numeric
    In [2]: type(Numeric.cos)
    Out[2]: <type 'builtin_function_or_method'>

  I see no way to associate a `unique` path to these objects

- All these don't work because they are builtin's
  (i.e, usually defined in some .so)

from pynotes import *
import math
addnote(math.cos)

import Numeric
addnote(Numeric.cos)

from scipy import xplt
addnote(xplt.plg)


These work:

from scipy import optimize
addnote(optimize)
notehelp(optimize)

addnote(optimize.newton)
notehelp(optimize.newton)

"""


import inspect
import os
from types import BuiltinFunctionType
import re   # for regular expression search

EDITORCOMMAND="xemacs"
PYNOTEDIR=os.path.expanduser("~/.pynote")
PYNOTEDIRS=["/usr/share/doc/python2.3-scipy/pynotes/", PYNOTEDIR]
# This is list of dirs to search for additional docs.
# By this it would be possible to provide updated docs with
# each release of scipy




def construct_filename(obj):
    """Construct a (hopefully) unique filename
       associated with the object.

       Technical comments:
       This is easy if inspect.getsourcefile(<obj>) works
       of   <object>.__file__ is available.

       Example:
         import math
         math.__file__

       However, for the builtin ones this is less obvious:
         math.cos?
       Fortunately then
         math.cos.__module__
       exists.


       Hmm, that does not work for Numeric ufuncs:
       For example
       import Numeric
       Numeric?
       Numeric.cos.<TAB>

     """


##    # FIXME: I don't know the type of ufuncs ;-(
##    # FIXME: (and even if so, I don't know how to handle them)
##    if type(obj)=='ufunc':
##        print "We can't handle 'ufunc'"
##        return None


    try:
        objfile=inspect.getsourcefile(obj)
    except TypeError:
        print "We can't handle ", type(obj)
        return None

    # for example for
    #    import math
    #    print inspect.getsourcefile(math)
    # we get None
    if objfile==None:
        if hasattr(obj,"__file__"):
            objfile=obj.__file__
        else:
            print "unable to find the path ..."
            return None
    return objfile


def objpath2notefile(objfile,pynotedir=PYNOTEDIR):
    """Convert the path for a given object to
       the corresponding notefile
    """
    # Now we have for a given object its path, e.g.
    # /usr/lib/python2.3/lib-dynload/math.so
    # One could split off the initial "/usr/lib/python2.3/".
    # However, this assumes that all modules are
    # located there.
    # So we choose this variant, which
    # leads to longer paths, but (hopefully) unique ones.
    if objfile[0]=="/":
         fullpath=os.path.join(pynotedir,objfile[1:])
    else:
        print "FIMXE: don't know how to handle paths which"
        print "don't start with / (i.e. non-Unix ...)"
        sys.exit(0)

    # split extension and replace by ".txt"
    return os.path.splitext(fullpath)[0]+".txt"


def addnote(obj):
    """Add a note to the directory ~/.pynote"""

    objfile=construct_filename(obj)
    if objfile==None:
        print "sorry, no path to object found .."
        return


    notefile=objpath2notefile(objfile)
    notepath=os.path.split(notefile)[0]
    if not os.path.exists(notepath):
        os.makedirs(notepath)       # recursive makedir

    os.system(EDITORCOMMAND+" "+notefile)



def notehelp(obj):
    """Routine which does provide help on the given object.

       No formatting is done, just a bare print of the doc-string.
       Additionally it is checked for local documentation
       which is displayed.

       Notice that all directories in the list PYNOTEDIRS
       are checked. This allows to supply
       global additions to scipy.
    """
    print inspect.getdoc(obj)
    #check if there is maybe an additional note
    objpath=construct_filename(obj)
    if objpath!=None:  # we successfully constructed a path
        for notedir in PYNOTEDIRS:
            notefile=objpath2notefile(objpath,notedir)
            if os.path.exists(notefile):
                #print "FILE:", notefile
                print "----- additional note ---------"
                #print "====== ",notefile
                for lines in open(notefile):
                    print lines,
                print
                print "-------------------------------"


def _getallnotefiles():
    """Return list of all .txt files recursively found in PYNOTEDIRS."""
    notefiles=[]

    def addfiles(notefiles, dirname, fnames):
        #print dirname,fnames
        for file in fnames:
            full_file=os.path.join(dirname,file)
            if (os.path.isfile(full_file)
                and os.path.splitext(file)[1]==".txt"):
                notefiles.append(full_file)

    for notedir in PYNOTEDIRS:
        os.path.walk(notedir,addfiles,notefiles)
    return notefiles


def shownotes():
    """Show all notes in ~/.pynote"""

    print "Available notes:"
    for file in _getallnotefiles():
        if os.path.isfile(file):
            print file


def findnotes(regexp):
    """Search for `regexp` in all notes in ~/.pynote"""

    flags=re.LOCALE|re.IGNORECASE
    pattern=re.compile(regexp,flags)

    found=0
    for file in _getallnotefiles():
        if os.path.isfile(file) and os.path.splitext(file)[1]==".txt":
            #print file
            fp=open(file)
            for line in fp.readlines():
                #print line
                result=pattern.search(line)
                if result:
                    print file
                    print "  ",line
                    break
                found=1
            fp.close()

    if found==0:
        print "No match found"




if __name__=="__main__":
    # some simple tests:
    import Numeric
    addnote(Numeric)
    notehelp(Numeric)

    import scipy
    print "use `super` in the note, so it will be found"
    addnote(scipy)
    notehelp(scipy)

    print "Show all available notes:"
    shownotes()

    print "Search example:"
    findnotes("super")



















More information about the Scipy-dev mailing list