[Numpy-discussion] Numpy/Cython Google Summer of Code project idea
Sat Mar 8 04:15:17 CST 2008
On Fri, Mar 7, 2008 at 8:10 AM, Joris De Ridder
> Thanks. I've a few questions concerning the objections against ctypes.
> It's part of the Python standard library, brand new from v2.5, and it
> allows creating extensions. Disregarding it, requires therefore good
> arguments, I think. I trust you that there are, but I would like to
> understand them better. For ctypes your extensions needs to be
> compiled as a shared library, but as numpy is moving towards Scons
> which seem to facilitate this quite a lot, is this still a difficulty/
> objection? Secondly, looking at the examples given by Travis in his
> Numpy Book, neither pyrex nor ctypes seem to be particularly user-
> friendly concerning Numpy ndarrays (although ctypes does seem slightly
> easier). From your email, I understand it's possibly to mediate this
> for Cython. From a technical point of view, would it also be possible
> to make ctypes work better with Numpy, and if yes, do you have any
> idea whether it would be more or less work than for Cython?
As Chris B. said, I also think that ctypes and cython are simply
different, complementary tools. Cython allows you to create complete
functions that can potentially run at C speed entirely, by letting you
bypass some of the more dynamic (but expensive) features of python,
while retaining a python-like sytnax and having to learn a lot less of
the Python C API. Ctypes is pure python, so while you can access
arbitrary shared libraries, it won't help you one bit if you need to
write new looping code and the execution speed in pure python isn't
enough. At that point if ctypes is your only tool, you'd need to
write a pure C library (to the pure Python C API, with manual memory
management included) and access it via ctypes.
The point we're trying to reach is one where most of the extension
code for numpy is Cython, to improve its long-term maintainability, to
make it easier for non-experts in the C API to contribute 'low level'
tools, and to open up future possibilities for fast code generation.
I don't want to steal Travis' thunder, but I've heard him make some
very interesting comments about his long term ideas for novel tools to
express high-level routines in python/cython into highly efficient
low-level representations, in a way that code written explicitly to
the python C API may well make very difficult.
I hope this (Travis' ideas teaser and all :) provides some better
perspective on the recent enthusiasm regarding cython, as a tool
complementary to ctypes that could greatly benefit numpy and scipy.
If it doesn't it just means I did a poor job of communicating, so keep
on asking. We all really want to make sure that this is something
where we reach technical consensus; the fact that Sage has been so
successful with this approach is a very strong argument in favor (and
they've done LOTS of non-trivial work on cython to further their
goal), but we still need to ensure that the numpy/scipy community is
equally on board with the decision.
More information about the Numpy-discussion