[SciPy-dev] Inclusion of cython code in scipy

Travis E. Oliphant oliphant@enthought....
Thu Apr 24 08:57:03 CDT 2008


Stéfan van der Walt wrote:
> 2008/4/24 Prabhu Ramachandran <prabhu@aero.iitb.ac.in>:
>   
>>  Lets take a simple case of someone wanting to handle a growing
>>  collection of say a million particles and do something to them.  How do
>>  you do that in cython/pyrex and get the performance of C and interface
>>  to numpy?  Worse, even if it were possible, you'll still need to know
>>  something about allocating memory in C and manipulating pointers.  I can
>>  do that with C++ and SWIG today.
>>     
>
>   
Thanks for this example Stefan.   It helps to clarify the situation.   I 
don't have strong feelings right now either way and so I'm staying out 
of the discussion.   However, there are a few points I should respond to.

> The beauty of the Cython approach is that you
>
> a) Never have to worry about INCREF and DECREF
>   
This isn't quite true.   There are some NumPy C-API calls that Pyrex 
does not get right (e.g. recognizing that nearly all calls taking a 
PyArray_Descr object steal the reference to it), and so you have to 
manage it yourself.   This could perhaps be fixed in Cython
> b) Can use Python calls within C functions.  You don't want to do that
> in your fast inner loop, but take the example above: we only copy
> arrays infrequently, and then we'd like to have the full power of
> numpy indexing.  Suddenly, sorting, averaging, summing becomes a
> one-liner, just like in Python, at the expense of one Python call (and
> this won't affect execution time in the above example).
>
> c) Debug in a much cleaner way than C++ or C code: fewer memory leaks,
> introspection of source etc.
>   
I've also found this to be not quite true, either.  Every time I've done 
something serious with Pyrex (Cython),  I've had to look at the produced 
C-code to "get it right" and then it's only because I know the Python 
C-API and understand reference counting that I can "get it right."    
I'm not sure how debugging is easier, especially if the bug is a 
misunderstanding of how the code-generator actually works.  Perhaps the 
code-generator is improving and will get to the point where that is not 
true.    

There are a lot of nice things about Cython, however, lest you think I'm 
against what is being suggested.  I'm not.  I generally like the 
direction it is heading.  For example, the instrospection of source is 
useful, however (the way you can look at the Cython source code when 
getting help on a function). 

I would prefer it, however, if the ability to compile Python were done 
without changing the language.   It looks like Cython is moving in that 
direction, but it is not quite there yet. 

My ideal world would be the ability to emit machine instructions 
directly at various points in the Python code by just writing a 
restricted set of Python syntax.  Perhaps this could be seamless.   
Then, it would be much easier to just write Python code and not worry 
about it being "too slow" during critical loops.

-Travis





More information about the Scipy-dev mailing list