[Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...)

Lisandro Dalcin dalcinl@gmail....
Tue Apr 29 18:16:25 CDT 2008

David, I briefly took a look at your code, and I have a very, very
important observation.

Your implementation make uses of low level dlopening. Then, your are
going to have to manage all the oddities of runtime loading in the
different systems. In this case, 'libtool' could really help. I know,
it is GPL, but AFAIK it has some special licencing that let's you ship
it with your code in the same licence terms than your code.

But, I definitely think that a betther approach would be using a stubs
mechanism ala TCL, or wath is currently used in some extension modules
in core python, like cStringIO. In short, you access all your
functions from a pointer do a struct (statically or heap allocated)
where each struct member is filled with a pointer to a function. This
is pretty much similar to C++ virtual tables, or the Cython cdef's
classes with cdef's methods. And then you just let Python do the work
of dynamic loading of extension modules. The numpy C/API uses a
similar approach, but uses an array. IMHO, the struct approach is

What do you think about this?

On 4/28/08, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>     I've just started working on a prototype for a plugin system for
>  numpy. The plugin aims at providing a framework for the following user
>  cases:
>     - runtime selection of blas/lapack/etc...: instead of harcoding in
>  the binary one blas/lapack implementation, numpy could choose the SSE
>  optimized if the CPU supports SSE, etc...
>     - this could also be used for core numpy, for example ufuncs: if we
>  want to start implementing some tight loop with aggressively optimized
>  code (SSE, etc...), we could again ship with a default pure C
>  implementation, and choose the best one at runtime.
>     - we could even have a system to choose a different implementation
>  (for example, right now, scipy is shipped with a slow fft for licensing
>  issues mainly, and people installing fftw could then tell scipy to use
>  fftw instead of the included one).
>  Right now, the prototype does not do much, and only works for linux; I
>  mainly focused on automatic generation of the plugin from a list of
>  functions, and transparent use from numpy point of view. It provides the
>  plugin api through pure function pointers, without the need for the user
>  to be aware of it. For example, if you have an api with the following
>  functions:
>  void    foo1();
>  int     foo2();
>  int     foo3(int);
>  int     foo4(double* , double*);
>  int     foo5(double* , double*, int);
>  The current implementation would build the boilerplate to load those
>  functions, etc... and you would just use those functions in numpy like
>  the following:
>  init_foo();
>  /* all functions are prefixed with npyw, for numpy wrapper */
>  npyw_foo1();
>  npyw_foo2(n);
>  etc...
>  The code can be found there:
>  https://code.launchpad.net/~david-ar/+junk/numplug
>  And some thinking (pretty low content for now):
>  http://www.scipy.org/RuntimeOptimization
>  cheers,
>  David
>  _______________________________________________
>  Numpy-discussion mailing list
>  Numpy-discussion@scipy.org
>  http://projects.scipy.org/mailman/listinfo/numpy-discussion

Lisandro Dalcín
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

More information about the Numpy-discussion mailing list