[Numpy-discussion] a possible way to implement a plogin system

David Cournapeau cournapeau@cslab.kecl.ntt.co...
Wed Apr 30 22:03:46 CDT 2008


On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote:
> David, in order to put clear what I was proposing to you in previous
> mail regarding to implementing plugin systems for numpy, please take a
> look at the attached tarball.

Thanks for looking at this Lisandro.

The problem I see with the approach of struct vs direct function
pointers is of ABI compatibility: it is easy to mess things up with
structures. There is the advantage of using only one dlsym (or
equivalent) with the struct, which may be much faster than using
hundreds of dlsym for each function. Linux does not seem to have
problems with that, but mac os X for example felt slow when I tried
doing this for several thousand functions. I did not go really far on
that, though. I don't really see why using a struct is cleaner, though.
That's really the same thing (function pointers), in both cases the name
namespace pollution will be the same, and in both cases there will be a
need to generate source code.

> 
> IMHO, this is the easier and cleaner way to deal inside python with
> plugins written in low-level C. It does not depend explicitely on
> 'dlopen' stuff from your side.

Concerning the loading mechanism, I don't understand the point of using
PyCObject_Import. By quickly looking at the code of the function, the
plugin needs to be a python module in that case, which is not needed in
our case; I don't like the fact that it is not documented either.

The code for dlopen/dlclose is really short. For each platform, it is
like 50 lines of code, and we have a better control on what we can do
(which may be needed; for example, you want want to load symbols from a
dll A, but the symbols are only in dll B, which dll A depends on; that's
a problem that does not happen for python extensions, I think; I don't
really know to be honest).

The one thing which may have been useful by using python mechanism is
the binary location detection, but that won't be needed in our case,
since the plugins will always be distributed at the same time as
numpy/scipy, hence we will have tight control on their location.

> It even works on
> systems with no dynload, provided that all is put inside the python
> executable.

That won't be needed: the point is to have several implementations with
the same functions. If you link everything statically, you will get name
clashes anyway (if you want to stay portable). On those platforms,
plugins should be disabled when building numpy/scipy.

cheers,

David



More information about the Numpy-discussion mailing list