[Numpy-discussion] Question about Optimization (Inline and Pyrex)

Francesc Altet faltet@carabos....
Tue Apr 17 14:00:49 CDT 2007

El dt 17 de 04 del 2007 a les 16:43 +0000, en/na Simon Berube va
> I recently made the switch from Matlab to Python and am very
> interested in optimizing certain routines that I find too slow in
> python/numpy (long loops).
> I have looked and learned about the different methods used for such
> problems such as blitz, weave and pyrex but had a question for more
> experienced developpers.
> It appears that pyrex is the fastest of the bunch with weave very
> close behind but at the same time pyrex requires entirely different
> modules while weave can be inserted almost painlessly into existing
> code. Is the speed gain and usefulness of pyrex severely limited by
> the extra maintenance required y having separate "fast" routines from
> the rest of the code files?
> I am greatly interested in finding out what more experienced
> developers feel about these issues given that I may be completely off
> track and missing on a useful tool(pyrex) thinking weave is better
> than it actually is and I am quite frankly afraid of writing routines
> in one format and realizing later that it creates problems that I need
> to rewrite.
> I have tried searching for previous similar posts but could not find
> any. My apologies if this is a repeat or a severly dumb question.

Well, this is a delicate question. Let me put something clear before.
Pyrex code *might* be fast (as fast as C code can be in fact) if you
write good code, which is not an easy thing in most of situations mainly
because this does require mastery of not only the Pyrex language (which,
due to its similarity to Python, is relatively simple to learn), but
also (and specially) the internals of how your machine architecture
works (CPU bottlenecks, memory bottlenecks...).

When you compare weave (or whatever) against Pyrex in numerical
computations, you should have in mind other features as well and
specially the easy of use and the convenience to access the elements of
your numerical objects. I'm not a weave user, but I know that it allows
merging the weave code in your Python code and besides allows
multidimensional indexing (Pyrex don't). So, generally speaking, weave
is more high level (but still, fast!) than C, Fortran or Pyrex for doing
this kind of computations and depending on your needs, these factors
(and not only speed) can be worth considering.

Having said that, if you need to get all the performance that you
platform can offer to you, then Pyrex is an excellent option in that it
allows getting the maximum performance (if well coded, of course) from
the inside of the language. In addition, as it is heavily based on
Python syntax, it allows object oriented programming and excellent
interaction with Python code. These aforementioned factors are normally
very important ones when you have to develop relatively large modules
with high efficiency in mind. However, it must be stressed that Pyrex
*doesn't* allow to access multidimensional data in a convenient way (you
need to compute the indices your own for accessing the flat data array
in memory). It is true that this should'nt be a handicap for
undimensional or two-dimensional data, but it can be a pain if most of
your code has to deal with highly multidimensional objects.

Finally, don't let benchmarks fool you. If you can, it is always better
to run your own benchmarks made of your own problems. A tool that can be
killer for one application can be just mediocre for another (that's
somewhat extreme, but I hope you get the point).

Hope that helps,

Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth

More information about the Numpy-discussion mailing list