[Numpy-discussion] Memory hungry reduce ops in Numpy

Bruce Southey bsouthey@gmail....
Tue Nov 15 09:28:47 CST 2011


On 11/14/2011 10:05 AM, Andreas Müller wrote:
> On 11/14/2011 04:23 PM, David Cournapeau wrote:
>> On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller
>> <amueller@ais.uni-bonn.de>   wrote:
>>> Hi everybody.
>>> When I did some normalization using numpy, I noticed that numpy.std uses
>>> more ram than I was expecting.
>>> A quick google search gave me this:
>>> http://luispedro.org/software/ncreduce
>>> The site claims that std and other reduce operations are implemented
>>> naively with many temporaries.
>>> Is that true? And if so, is there a particular reason for that?
>>> This issues seems quite easy to fix.
>>> In particular the link I gave above provides code.
>> The code provided only implements a few special cases: being more
>> efficient in those cases only is indeed easy.
> I am particularly interested in the std function.
> Is this implemented as a separate function or an instantiation
> of a general reduce operations?
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
The'On-line algorithm' 
(http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm) 
<http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm> 
could save you storage. I would presume if you know cython that you can 
probably make it quick as well (to address the loop over the data).

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20111115/dcd2f772/attachment-0001.html 


More information about the NumPy-Discussion mailing list