[Numpy-discussion] NumPy re-factoring project

Sturla Molden sturla@molden...
Thu Jun 10 12:17:08 CDT 2010

Another suggestion I'd like to make is bytearray as memory buffer for 
the ndarray. An ndarray could just store or extend a bytearray, instead 
of having to deal with malloc/free and and the mess that comes with it. 
Python takes care of the reference counts for bytearrays, and ctypes 
calls the NumPy core DLL. Yes it will not work with older versions of 
Python. But for a complete refactoring ("NumPy 3000"), backwards 
compatibility should not matter much. (It's easier to backport bytearray 
than track down memory leaks in NumPy's core.)


Den 10.06.2010 18:48, skrev Sturla Molden:
> I have a few radical suggestions:
> 1. Use ctypes as glue to the core DLL, so we can completely forget 
> about refcounts and similar mess. Why put manual reference counting 
> and error handling in the core? It's stupid.
> 2. The core should be a plain DLL, loadable with ctypes. (I know David 
> Cournapeau and Robert Kern is going to hate this.) But if Python can 
> have a custom loader for .pyd files, so can NumPy for it's core DLL. 
> For ctypes we just need to specify a fully qualified path to the DLL, 
> which can be read from a config file or whatever.
> 3. ctypes will take care of all the mess regarding the GIL. Again 
> there is no need to re-invent the wheel. ctypes.CDLL releases the GIL 
> when calling into C, and re-acquires before making callbacks to 
> Python. ctypes.PyDLL keeps the GIL for legacy library code that are 
> not threadsafe.
> ctypes will also make porting to other Python implementations easier 
> (or even other languages: Ruby, JacaScript) easier. Not to mention 
> that it will make NumPy impervious to changes in the Python C API.
> 4. Write the core in C++ or Fortran (95/2003), not C. ANSI C (aka 
> C89). Use std::vector<> instead of malloc/free for C++, and possibly 
> templates for generics on various dtypes.
> 5. Allow OpenMP pragmas in the core. If arrays are above a certain 
> size, it should switch to multi-threading.
> Sturla
> Den 10.06.2010 15:26, skrev Charles R Harris:
>> A few thoughts came to mind while reading the initial writeup.
>> 1) How is the GIL handled in the callbacks.
>> 2) What about error handling? That is tricky to get right, especially 
>> in C and with reference counting.
>> 3) Is there a general policy as to how the reference counting should 
>> be handled in specific functions? That is, who does the reference 
>> incrementing/decrementing?
>> 4) Boost has some reference counted pointers, have you looked at 
>> them? C++ is admittedly a very different animal for this sort of 
>> application.
>> Chuck
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20100610/3b70d440/attachment.html 

More information about the NumPy-Discussion mailing list