[Numpy-discussion] is numerical python only for prototyping ?

[Perry Greenfield]

On Jul 26, 2005, at 12:41 PM, Sebastian Haase wrote:

> Hi,
> This is not sopposed to be an evil question; instead I'm hoping for the
> answer: "No, generally we get >=95% the speed of a pure C/fortran
> implementation" ;-)
> But as I am the strongest Python/numarray advocate in our group I get 
> often
> the answer that Matlab is (of course) also very convenient but 
> generally
> memory handling and overall execution performance is so bad that for 
> final
> implementation one would generally have to reimplement in C.
> We are a bio-physics group at UCSF developping new algorithms for
> deconvolution (often in 3D). Our data sets are regularly bigger than 
> several
> 100MB.  When deciding for numarray I was assuming that the "Hubble 
> Crowd" had
> a similar situation and all the operations are therefore very much 
> optimized
> for this type of data.
> Is 95% a reasonable number to hope for ?  I did wrap my own version of 
> FFTW
> (with "plan-caching"), which should give 100% of the C-speed. But 
> concerns
> arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c 
> are
> each 3D-datastacks creation of temporary data-arrays for 'c*a' AND 
> then also
> for 'b+...' would have to be very costly. (I think this is at least 
> happening
> for Numeric - I don't know about Matlab and numarray)
>

Is it speed or memory usage you are worried about? Where are you 
actually seeing unacceptable performance?

Offhand, I would say the temporaries are not likely to be serious speed 
issues (unless you running out of memory). We did envision at some 
point (we haven't done it yet) of recognizing situations where the 
temporaries could be reused.

As for 95% speed, it's not what required for our work (I think that 
what an acceptable speed ratio is depends on the problem). In many 
cases being within 50% is good enough except for heavily used things 
where it would need to be faster. But yes, we do plan on (and do indeed 
actually use it now) for large data sets where speed is important. We 
generally don't compare the numarray speed to C speed all the time (if 
we were writing C equivalents every time, that would defeat the purpose 
of using Python :-). Perhaps you could give a more specific example 
with some measurements? I don't think I would promise anyone that all 
one's code could be done in Python. Undoubtedly, there are going to be 
some cases where coding in C or similar is going to be needed. I'd just 
argue that Python let's you keep as much as possible in a higher level 
language and a little as necessary in a low level language such as C.

Perry