[Numpy-discussion] numpy FFT memory accumulation

Anne Archibald peridot.faceted@gmail....
Wed Oct 31 17:50:38 CDT 2007


On 31/10/2007, Ray S <subscriber100@rjs.org> wrote:
> I am using
> fftRes = abs(fft.rfft(data_array[end-2**15:end]))
> to do running analysis on streaming data. The N never changes.
> It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec.
> (Interestingly, Numeric FFT accumulates much slower..)
> (Commenting out that one line stops memory growth.)
>
> What can one do to alleviate this?
> Can I del() some numpy object or such?
> It's a bit of an issue for a program that needs to run for weeks.
>
> It's purpose is simply to argmax() the largest bin, which always
> falls within a small range - do I have another, better option than
> fft?

If the range is *really* small, you can try using a DFT - sometimes
that is fast enough, and gives you just the bins you're curious about.

If the range is bigger than that, but still a small fraction of the
FFT size, you can do some tricks where you band-pass filter the data
(with a FIR filter, say), downsample (which aliases frequencies
downward), and then FFT. Concretely, if your data is sampled at
80ksamp/s and you're taking 160ksamp FFTs, you're getting the spectrum
up to 35 kHz with a resolution of 0.5 Hz. If all you care about is the
spectral region from 5 kHz to 6 kHz but you still need the 0.5 Hz
spectral resolution, you can run a band-pass filter to throw out
everything outside (say) 4 to 7 kHz, then downsample to 8 ksamp/s; the
4 to 7kHz region will appear mirrored in the 1 to 4 kHz region of the
new FFTs, which will each be a tenth the size.

There are also "zoom fft" and "chirp-z" techniques which are supposed
to give you only part of the FFT, but the wisdom is that unless you
want less than a few percent of the data points you're better just
FFTing and throwing lots of data away.

Anne


More information about the Numpy-discussion mailing list