[Numpy-discussion] numpy FFT memory accumulation

Ray S subscriber100@rjs....
Thu Nov 1 14:38:52 CDT 2007


At 09:00 AM 11/1/2007, you wrote:
I saw that Numeric did also (I still use Numeric for smaller array
speed) but much more slowly.
I will try to repeat with a small demo and post.

It turns out to be some aspect of mixing numpy and Numeric;

the attached *Stable.py files allocate memory that stays exactly the 
same

mixedGrows.py grows by about 50kB/s while changing slightly, 
randomly; note that the array is assigned as numpy and operated on 
with Numeric functions.

My main app also uses mmap/numpy to assign the arrays to shared 
memory for multiple processes, and does other operations as well, 
possibly accounting for the faster memory growth.
I'll re-factor to an all-numpy solution and test.

I used Numeric functions for the ~40% speed increase, but I don't 
have a compiled version of arrayfrombuffer() for Numeric
http://www.canonical.org/~kragen/sw/arrayfrombuffer/ (compiler 
version woes)
so, I must use numpy.frombuffer() with mmap to create the shared 
arrays...

Ray
-------------- next part --------------
import numpy
import numpy.fft as fft

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
numpy.random.seed()

dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32)
while 1:
    fftArray = abs(fft.rfft(dataArray[:LARGE_FFT_SIZE]))
    fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC
    maxBin = numpy.argmax(fftArray[1000:5000])+1000
    print "\r%d" % (maxBin),
    dataArray[0:CIRC_BUFFER_SIZE] = numpy.random.binomial(5000000, .5, (CIRC_BUFFER_SIZE,))
    

-------------- next part --------------
import RandomArray
import Numeric
import FFT

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
RandomArray.seed()

def freqAnalysis():
    global infoArray,dataArray
    fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE]))
    maxBin = Numeric.argmax(fftArray[1000:5000])+1000
    return maxBin

def getMoreData():
    global infoArray,dataArray
    dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.binomial(5000000, .5, (CIRC_BUFFER_SIZE,))
    
## small array to hold shared variables
## 0 -> bufferInsertPos ## 1 -> dataTotSamples
## 2 -> sampPerRot ## 3 -> run signal
infoArray = Numeric.zeros((4,), Numeric.Float32)
dataArray = Numeric.zeros((CIRC_BUFFER_SIZE*2,), Numeric.Int32)
while 1:
    infoArray[2] = freqAnalysis() ## sampPerRot
    print "\r%d" % (infoArray[2]),
    getMoreData()
    

-------------- next part --------------
import RandomArray
import Numeric
import FFT
import mmap
import numpy

CIRC_BUFFER_SIZE = 2**16
LARGE_FFT_SIZE = 2048*8
RandomArray.seed()

dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32)
while 1:
    fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE]))
    fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC
    maxBin = Numeric.argmax(fftArray[1000:5000])+1000
    print "\r%d" % (maxBin),
    dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.normal(5000000, 2000000, (CIRC_BUFFER_SIZE,))
    



More information about the Numpy-discussion mailing list