[SciPy-dev] Inclusion of cython code in scipy
Travis E. Oliphant
Thu Apr 24 08:57:03 CDT 2008
Stéfan van der Walt wrote:
> 2008/4/24 Prabhu Ramachandran <firstname.lastname@example.org>:
>> Lets take a simple case of someone wanting to handle a growing
>> collection of say a million particles and do something to them. How do
>> you do that in cython/pyrex and get the performance of C and interface
>> to numpy? Worse, even if it were possible, you'll still need to know
>> something about allocating memory in C and manipulating pointers. I can
>> do that with C++ and SWIG today.
Thanks for this example Stefan. It helps to clarify the situation. I
don't have strong feelings right now either way and so I'm staying out
of the discussion. However, there are a few points I should respond to.
> The beauty of the Cython approach is that you
> a) Never have to worry about INCREF and DECREF
This isn't quite true. There are some NumPy C-API calls that Pyrex
does not get right (e.g. recognizing that nearly all calls taking a
PyArray_Descr object steal the reference to it), and so you have to
manage it yourself. This could perhaps be fixed in Cython
> b) Can use Python calls within C functions. You don't want to do that
> in your fast inner loop, but take the example above: we only copy
> arrays infrequently, and then we'd like to have the full power of
> numpy indexing. Suddenly, sorting, averaging, summing becomes a
> one-liner, just like in Python, at the expense of one Python call (and
> this won't affect execution time in the above example).
> c) Debug in a much cleaner way than C++ or C code: fewer memory leaks,
> introspection of source etc.
I've also found this to be not quite true, either. Every time I've done
something serious with Pyrex (Cython), I've had to look at the produced
C-code to "get it right" and then it's only because I know the Python
C-API and understand reference counting that I can "get it right."
I'm not sure how debugging is easier, especially if the bug is a
misunderstanding of how the code-generator actually works. Perhaps the
code-generator is improving and will get to the point where that is not
There are a lot of nice things about Cython, however, lest you think I'm
against what is being suggested. I'm not. I generally like the
direction it is heading. For example, the instrospection of source is
useful, however (the way you can look at the Cython source code when
getting help on a function).
I would prefer it, however, if the ability to compile Python were done
without changing the language. It looks like Cython is moving in that
direction, but it is not quite there yet.
My ideal world would be the ability to emit machine instructions
directly at various points in the Python code by just writing a
restricted set of Python syntax. Perhaps this could be seamless.
Then, it would be much easier to just write Python code and not worry
about it being "too slow" during critical loops.
More information about the Scipy-dev