[Numpy-discussion] Numarray header PEP

Todd Miller jmiller at stsci.edu
Wed Jun 30 14:55:04 CDT 2004


On Tue, 2004-06-29 at 17:32, gerard.vermeulen at grenoble.cnrs.fr wrote: 
> On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote
> > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote:
> > > > 
> > > > The PEP is attached.  It is formatted using the docutils package which
> > > > can be used to generate HTML or PDF.  Comments and corrections would be
> > > > appreciated.
> > > > 
> > > PyQwt is a Python extension which can be conditionally compiled against
> > > Numeric and/or numarray (both, one of them or none).
> > 
> > Well that's cool!  I'll have to keep the PyQwt guys in mind as potential
> > first users.
> > 
> > > Your PEP excludes importing of Numeric and numarray in the same C-extension.
> > 
> > This is true but I don't understand your solution so I'll blather on
> > below.
> > 
> > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray()
> > > and import_array() into a few functions with numarray specific names, so
> > > that the following becomes possible:
> > > 
> > > #include <Numeric/meta.h>
> > > /* defines the functions (no macros)
> > >    int Numeric_Present();
> > >    int Numeric_isArray();
> > >    void import_numeric();
> > >    to hide the Numeric C-API stuff in a small Numeric/meta.c file.
> > > */
> > > #include <numarray/meta.h>
> > > /* defines the functions (no macros)
> > >    int numarray_Present();
> > >    int numarray_isArray();
> > >    void import_numarray();
> > >    to hide the numarray C-API stuff in a small numarray/meta.c file.
> > > */
> > >
> > 
> > I may have come up with the wrong scheme for the Present() and
> > isArray().  With my scheme, they *have* to be macros because the API
> > functions are unsafe:  when numarray or Numeric is not present, the API
> > function table pointers are NULL so calling through them results in
> > either a fatal error or a segfault.
> >
> The macros can be hidden from the module file scope by wrapping them
> in a function (see attached demo)

Your demo is very clear... nice!

> > There is an additional problem that the "same functions" need to be
> > called through different API pointers depending on whether numarray 
> > or Numeric is being supported.  Redefinition of typedefs and enumerations
> > 
> > (or perhaps conditional compilation short-circuited re-definitions) may
> > also present a problem with compiling (or worse, running).
> >
> Tested and works. 
> >
> > I certainly like the idea of supporting both in the same extension
> > module,  but don't see how to get there, other than with separate
> > compilation units.  With seperate .c files,  I'm not aware of a problem
> > other than lots of global symbols.  I haven't demoed that yet so I am
> > interested if someone has made it work.
> > 
> Yes, you cannot mix the numarray API and Numeric API in the same .c
> file, but nevertheless you can hide the macros in small functions so
> that the macros don't pollute.

So... you use the "meta" code to provide package specific ordinary
(not-macro-fied) functions to keep the different versions of the
Present() and isArray() macros from conflicting.

It would be nice to have a standard approach for using the same
"extension enhancement code" for both numarray and Numeric.  The PEP
should really be expanded to provide an example of dual support for one
complete and real function, guts and all, so people can see the process
end-to-end;  Something like a simple arrayprint.  That process needs to
be refined to remove as much tedium and duplication of effort as
possible.  The idea is to make it as close to providing one
implementation to support both array packages as possible.  I think it's
important to illustrate how to partition the extension module into
separate compilation units which correctly navigate the dual
implementation mine field in the easiest possible way.

It would also be nice to add some logic to the meta-functions so that
which array package gets used is configurable.  We did something like
that for the matplotlib plotting software at the Python level with the
"numerix" layer, an idea I think we copied from Chaco.  The kind of
dispatch I think might be good to support configurability looks like
this:

PyObject *
whatsThis(PyObject *dummy, PyObject *args)
{
    PyObject *result, *what = NULL;
    if (!PyArg_ParseTuple(args, "O", &what))
      return 0;
    switch(PyArray_Which(what)) {
      USE_NUMERIC:
         result = Numeric_whatsThis(what); break;
      USE_NUMARRAY:
         result = Numarray_whatsThis(what); break;
      USE_SEQUENCE:
         result = Sequence_whatsThis(what); break;
    }
    Py_INCREF(Py_None);
    return Py_None;
}

In the above,  I'm picturing a separate .c file for Numeric_whatsThis
and for Numarray_whatsThis.  It would be nice to streamline that to one
.c and a process which somehow (simply) produces both functions.  

Or, ideally, the above would be done more like this:

PyObject *
whatsThis(PyObject *dummy, PyObject *args)
{
    PyObject *result, *what = NULL;
    if (!PyArg_ParseTuple(args, "O", &what))
       return 0;
    switch(Numerix_Which(what)) {
       USE_NUMERIX:
          result = Numerix_whatsThis(what); break;
       USE_SEQUENCE:
          result = Sequence_whatsThis(what); break;
    }
    Py_INCREF(Py_None);
    return Py_None;
}

Here, a common Numerix implementation supports both numarray and Numeric
from a single simple .c.  The extension module would do "#include
numerix/arrayobject.h" and "import_numerix()" and otherwise just call
PyArray_* functions.

The current stumbling block is that numarray is not binary compatible
with Numeric... so numerix in C falls apart.  I haven't analyzed every
symbol and struct to see if it is really feasible... but it seems like
it is *almost* feasible, at least for typical usage.

So, in a nutshell,  I think the dual implementation support you demoed
is important and we should work up an example and kick it around to make
sure it's the best way we can think of doing it.  Then we should add a
section to the PEP describing dual support as well.

Regards,
Todd





More information about the Numpy-discussion mailing list