[Numpy-discussion] Optimizing mean(axis=0) on a 3D array

Tim Hochberg tim.hochberg at ieee.org
Sun Aug 27 10:36:56 CDT 2006


Martin Spacek wrote:
> Tim Hochberg wrote:
>
>   
>> Here's an approach (mean_accumulate) that avoids making any copies of 
>> the data. It runs almost 4x as fast as your approach (called baseline 
>> here) on my box. Perhaps this will be useful:
>>
>>     
> --snip--
>   
>> def mean_accumulate(data, indices):
>>     result = np.zeros([32, 32], float)
>>     for i in indices:
>>         result += data[i]
>>     result /= len(indices)
>>     return result
>>     
>
> Great! I got a roughly 9x speed improvement using take() in combination 
> with this approach. Thanks Tim!
>
> Here's what my code looks like now:
>
>  >>> def mean_accum(data):
>  >>>     result = np.zeros(data[0].shape, np.float64)
>  >>>     for dataslice in data:
>  >>>         result += dataslice
>  >>>     result /= len(data)
>  >>>     return result
>  >>>
>  >>> # frameis are int64
>  >>> frames = data.take(frameis.astype(np.int32), axis=0)
>  >>> meanframe = mean_accum(frames)
>
> I'm surprised that using a python for loop is faster than the built-in 
> mean method. I suppose mean() can't perform the same in-place operations 
> because in certain cases doing so would fail?
>   
I'm not sure why mean is slow here, although possibly it's a locality 
issue -- mean likely computes along axis zero each time, which means 
it's killing the cache -- and on the other hand the accumulate version 
is cache friendly.  One thing to keep in mind about python for loops is 
that they are slow if you are doing a simple computation inside (a 
single add for instance). IIRC, they are 10's of times slower. However, 
here one is doing 1000 odd operations in the inner loop, so the loop 
overhead is minimal.

(What would be perfect here is something just like take, but that 
returned an iterator instead of a new array as that could be done with 
no copying -- unfortunately such a beast does not exist as far as I know)

I'm actually surprised that the take version is faster than my original 
version since it makes a big  ol' copy. I guess this is an indication 
that indexing is more expensive than I realize. That's why nothing beats 
measuring!

An experiment to reshape your data so that it's friendly to mean 
(assuming it really does operate on axis zero) and try that. However, 
this turns out to be a huge pesimization, mostly because take + 
transpose is pretty slow.

-tim


> Martin
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   






More information about the Numpy-discussion mailing list