[Numpy-discussion] Openmp support (was numpy's future (1.1 and beyond): which direction(s) ?)

David Cournapeau david@ar.media.kyoto-u.ac...
Mon Mar 24 12:37:38 CDT 2008


Matthew Brett wrote:
>
> I'm +3 for the plugin idea - it would have huge benefits for
> installation and automatic optimization.  What needs to be done?  Who
> could do it?

The main issues are portability, and reliability I think. All OS 
supported by numpy have more or less a dynamic library loading support 
(that's how python itself works, after all, unless you compile 
everything statically), but the devil is in the details. In particular:
    - I am not sure whether plugin unloading is supported by all OS (Mac 
os X posix api does not enable unloading, for example, you have to use a 
specific API which I do not know anything about). Maybe we do not need 
it, I don't know; unloading sounds really difficult to support reliably 
anyway (how to make sure numpy won't use the functions of the plugins ?)
    - build issues: it is really the same thing than ctypes at the build 
level, I think
    - an api so that it can be used throughout numpy. there should be a 
clear interface for the plugins, which does not sound trivial either. 
That's the most difficult part in my mind (well, maybe not difficult, 
but time-consuming at least).

That's one of the reason why I was thinking about a gradual move of most 
"core functionalities of the core" toward a separate C library, with a 
simple and crystal clear interface, without any reference to any python 
API, just plain C with plain pointers. We could then force this core, 
"pure" C library to be used only through dereferencing of an additional 
pointer, thus enabling dynamic change of the actual functions (at least 
when numpy is started).

I have to say I really like the idea of more explicit separation between 
the actual computation code and the python wrapping; it can only help if 
we decide to write some of the wrappers in Cython/ctypes/whatever 
instead of pure C as today. It has many advantages in terms of 
reliability, maintainability and performance (testing performance would 
be much easier I think, since it could be done in pure C).

cheers,

David


More information about the Numpy-discussion mailing list