[SciPy-dev] Thoughts on weave improvements

Pearu Peterson pearu at cens.ioc.ee
Mon Feb 11 13:22:59 CST 2002


First, let me say that I appreciate very much the idea of PyCOD and what
it tries to accomplish. Nevertheless, I find that a bucket of cold water
is in order. I hope it will be constructive ;-)

Ok, I can see few issues about accelerating Python with PyCOD or by
any other (already available) means:

1) About C++ extensions that PyCOD would generate. Why not C? Compiling
C++ can be very time and memory consuming task when compared to C or
Fortran compilation. For example, the gain from weave (as tested with
weave tests) was not so remarkable on my PII,96MB laptop until I upgraded
the memory to 160MB and C++ could run without swapping. I find this issue
quite serious drawback also when using weave, especially when developing

2) About PyCOD compiling Python functions to extension functions. If these
Python functions are not simple (they may use 3rd party modules, complex
Python features like classes, etc) then in order PyCOD to be applicable in
these (and the most practical) cases, it should be able to transform _any_
Python code to C. (There was a project that translated python code to C
code, anyone knows what is the state of this project?) Anyway, I find this
task rather difficult if not possible to accomplish in a reasonable amount
of time. Otherwise, if PyCOD cannot do that, then I am afraid that it
will be a toy program in scipy ;)

3) About callbacks. Indeed, when using 'some hairy, expensive python code'
from C/C++/Fortran, most of the time is spent in Python (as Prabhu tests
show, pure Python is approx 1000 times slower than any compiled language
mentioned above). However, this 'some hairy, expensive python code' need
not to be pure Python code, it may contain calls to extension modules that
will speed up these callback functions. So, I don't think that
calling callbacks from C/C++/Fortran would be remarkably expensive unless
these callbacks will be really simple (true, sometimes they just are).

On Fri, 8 Feb 2002, Pat Miller wrote:


> It might be better if the integration routine, using its knowledge that
> the argument to x must be a PyFloat (C++ double) could use a C++ 
> accelerated function instead of slow callbacks to Python.  Not important
> for a quick numeric integration, but crucial for using Python as a
> true C/FORTRAN replacement.

The last statement I find hard to believe. It is almost too trivial to
write down few reasons (don't let your customers to know about these ;-):
1) Python (or any other scripting language) cannot never compete with
C/FORTRAN in performance. (I truly hope that you can prove me to be
wrong here.)
2) There are too many high-performance and well-tested C/FORTRAN codes
available (atlas,lapack,fftw to name just few) and to repeat their
implementation in Python is unthinkable, in addition to the point 1).

Simple, transparent, and straightforward mixing of Python/C/C++/Fortran
languages seems to be the way to go in scientific computing within Python.


More information about the Scipy-dev mailing list