[SciPy-dev] Inclusion of cython code in scipy

Nathan Bell wnbell@gmail....
Wed Apr 23 11:49:22 CDT 2008


On Wed, Apr 23, 2008 at 11:00 AM, Brian Granger <ellisonbg.net@gmail.com> wrote:

>  1) The high-profile scientific computing+python projects that are
>  written using it (or pyrex):
>
>  mpi4py (mpi binding in python - currently being rewritten to use cython)
>  pytables (hdf5 python bindings)

These are wrappers.  Writing wrappers in Cython is a fine idea.

>  sage

Sage claims over 100K lines of Cython source code, so I agree, they do
have an interest in maintaining Cython.  They also have an interest in
getting others to sign on.

>  numpy.random

Doesn't this wrap the Mersenne Twister PRNG?

>  RIght now, the worst case scenario is if cython became unmaintained.
>  But, with the large number of people depending on it, I think the
>  community would simply pick up the slack - the beauty of open source.

I disagree.  It is rare for an open source project to reach a
sustainable level before the original authors depart.

>  2) Cython is unique in what it does.
>
>  All the other solutions for making python code fast (swig, f2py,
>  ctypes, boost, etc.) still require spending lots of time writing low
>  level C/C++/fortran code.  Sure, many of us have the skills to do
>  that, but the reality is that few of us have the _time_ to write that
>  code.  If I had to write the ipython distributed array package in
>  C/C++ and then wrap it into python it simply _would_not_happen_
>  (because of time constraints).

I don't see how writing Cython is easier than writing C/C++/Fortran.
Clearly if you only know Python, then Cython is an easier language to
learn.  However, I doubt the Cython abstraction is so complete that
one can avoid learning C.

You conflate the issues of writing code and wrapping it for Python.  I
agree that the latter is a painful excercise, even with tools like
SWIG.  I'd happily replace my SWIG wrappers with Cython if the need
arose.  However, in the context of larger-scale implementations, I
find it hard to believe that the claimed benefits of Cython outweigh
the concerns I've laid out.

>  True, a C/C++ library would live on _if_ these projects vanished.  But
>  I don't think you have made a strong argument that these projects will
>  in fact vanish.  Furthermore, I think your actions say the opposite.
>  You have invested a huge amount of time working on scipy.sparse (which
>  is greatly appreciated).  I could be wrong, but I don't think you
>  would do that if you really thought scipy was going to die soon :)

Clearly I think SciPy has a long life ahead of it.  At the same time,
I spent effort ensuring that sparsetools knows absolutely nothing
about NumPy/SciPy.  IMO the utility of a stand-alone library is
greater than a tighter coupling between sparsetools and Python.  For
instance, I can now combine the C++ I've written for PyAMG with
sparsetools to make a lightweight algebraic multigrid code in pure
C++.


>  Another relevant point:  recently I have spent quite a lot of time
>  wrapping templated C++ code with Cython (a 1 million LOC parallel c++
>  electromagnetics code).  This code uses the STL extensively and Cython
>  has worked really well.  If someone really wants to use cython in the
>  same way you have used swig, that is completely possible - you still
>  end up with the nice C++ library underneath.

As I've said before, using Cython as a wrapper generator is perfectly
reasonable.  Using Cython to implement the core functionality of that
1M LOC C++ library would be foolish.

>  The big difference though between cython and swig is that the top
>  level layer in swig is slow pure python code.  In cython, you get fast
>  C/python extension types that can be called from other C/C++ extension
>  code without going through a slow python layer.  That is a huge
>  difference.

Which support the case that Cython is a decent wrapper generator.

-- 
Nathan Bell wnbell@gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


More information about the Scipy-dev mailing list