[Numpy-discussion] numpy FFT memory accumulation

Ray S subscriber100@rjs....
Thu Nov 1 14:38:52 CDT 2007

At 09:00 AM 11/1/2007, you wrote:
I saw that Numeric did also (I still use Numeric for smaller array
speed) but much more slowly.
I will try to repeat with a small demo and post.

It turns out to be some aspect of mixing numpy and Numeric;

the attached *Stable.py files allocate memory that stays exactly the
same

mixedGrows.py grows by about 50kB/s while changing slightly,
randomly; note that the array is assigned as numpy and operated on
with Numeric functions.

My main app also uses mmap/numpy to assign the arrays to shared
memory for multiple processes, and does other operations as well,
possibly accounting for the faster memory growth.
I'll re-factor to an all-numpy solution and test.

I used Numeric functions for the ~40% speed increase, but I don't
have a compiled version of arrayfrombuffer() for Numeric
http://www.canonical.org/~kragen/sw/arrayfrombuffer/ (compiler
version woes)
so, I must use numpy.frombuffer() with mmap to create the shared
arrays...

Ray
-------------- next part --------------
import numpy
import numpy.fft as fft

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
numpy.random.seed()

dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32)
while 1:
fftArray = abs(fft.rfft(dataArray[:LARGE_FFT_SIZE]))
fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC
maxBin = numpy.argmax(fftArray[1000:5000])+1000
print "\r%d" % (maxBin),
dataArray[0:CIRC_BUFFER_SIZE] = numpy.random.binomial(5000000, .5, (CIRC_BUFFER_SIZE,))

-------------- next part --------------
import RandomArray
import Numeric
import FFT

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
RandomArray.seed()

def freqAnalysis():
global infoArray,dataArray
fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE]))
maxBin = Numeric.argmax(fftArray[1000:5000])+1000
return maxBin

def getMoreData():
global infoArray,dataArray
dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.binomial(5000000, .5, (CIRC_BUFFER_SIZE,))

## small array to hold shared variables
## 0 -> bufferInsertPos ## 1 -> dataTotSamples
## 2 -> sampPerRot ## 3 -> run signal
infoArray = Numeric.zeros((4,), Numeric.Float32)
dataArray = Numeric.zeros((CIRC_BUFFER_SIZE*2,), Numeric.Int32)
while 1:
infoArray[2] = freqAnalysis() ## sampPerRot
print "\r%d" % (infoArray[2]),
getMoreData()

-------------- next part --------------
import RandomArray
import Numeric
import FFT
import mmap
import numpy

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
RandomArray.seed()

dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32)
while 1:
fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE]))
fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC
maxBin = Numeric.argmax(fftArray[1000:5000])+1000
print "\r%d" % (maxBin),
dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.normal(5000000, 2000000, (CIRC_BUFFER_SIZE,))