[SciPy-User] frequency components of a signal buried in a noisy time domain signal
Sat Feb 27 06:49:19 CST 2010
On 27 February 2010 04:41, Nils Wagner <firstname.lastname@example.org> wrote:
> On Fri, 26 Feb 2010 16:32:01 -0500
> I am newbie to signal processing.
> Is there a good introduction that you can recommend ?
> There are so many books on signal processing. It should
> cover engineering applications.
Hmm. I don't have a good example in mind. For an overview you could
try Numerical Recipes (but keep in mind that there are free python
implementations of almost all the algorithms they describe, and there
are often better algorithms out there). Mostly I have used signal
processing in a scientific context, often somewhat different.
> What makes a signal weak/strong periodic ?
By an exactly periodic signal I mean something like sin(f*t). Such a
signal produces a very narrow peak of a characteristic shape in an
FFT, and so can be recognized even in the presence of quite strong
noise. If your signal is only quasi-periodic - perhaps the frequency
is a slowly-varying function of time - you'll have a much broader
peak, which will be much lower for the same input signal amplitude,
and hence more difficult to distinguish from noise. If your signal is
only quasi-periodic, or comes and goes in the data, you may want to do
a series of FFTs on short, overlapping pieces of the data, so you can
look at time evolution of the signal's spectral properties.
By weak versus strong, I just meant: is your signal so weak that
there's some question you'll be able to detect it at all, or is it
strong enough that you can try to extract more subtle properties?
> The signals come from real-life application (pressure /
> acceleration data).
> Do I need a filter before I apply FFT ?
The short answer is "probably not", or at least, not in the usual
sense of "filter". You will probably want to extend your data set so
that it is a power of two, just to make the FFT faster. Extending your
data set in time has the effect of interpolating the FFT, so it may
well be worth extending the data set further, to double or four times
its original length. This will broaden your peaks and affect the
statistics of the noise. (And, incidentally, if you're going to do
this, you should extend the array either by repeating the input values
or by filling the extra space with the mean of the input array to
avoid a spurious bump at low frequencies.)
Once you've got the FFT, you should know what different kinds of
signal look like. A pure sinusoid that's at one of the FFT frequencies
- which happens when there's exactly an integer number of turns in
your data - will prodduce a spike at exactly one of your FFT
frequencies. A pure sinusoid that's between two FFT frequencies will
produce a sin(x)/x response in nearby bins; in particular, a sinusoid
midway between two FFT frequencies will produce two high-power bins
that are substantially less high than the single bin would be if the
same signal were exactly at an FFT frequency; this is one reason
extending the data to interpolate the FFT can be a good idea.
There are also tricks that can be done once you've found a peak,
looking at the precise shape of the peak to determine whether the
periodic signal fills the whole of your data and if not where it is.
If you have a more broad-band signal, perhaps because it is only
approximately periodic, then you'll have an extended region of extra
power in the FFT. This can be a little tricky to recognize, if it's
not obvious from inspection; one thing to try is smoothing the power
spectrum (the squared amplitude of the FFT) so that local increases
show more prominently.
> What would you do if you know nothing about the origin of
> the signal ?
This rather depends on what I was looking for. But if I had a time
series in which I suspected some sort of periodicity, I'd start by
plotting the power spectrum and looking for peaks that stood out well
above the local average. I'd also try plotting a smoothed power
Also worth trying would be taking a series of overlapping FFTs each
covering part of the data; I might see a signal in some of these
individual FFTs, which would tell me where it was in the data, or I
might average all the power spectra (which is roughly equivalent to
smoothing a long-term FFT) looking for a broad-band signal.
If I expected a periodic signal that had harmonic content - a series
of pulses rather than a pure sinusoid, for example - I'd want to look
at the total power in several harmonics at once, adding the power at
f, 2f, 3f, 4f, ....
If I found an exactly periodic signal, I could investigate further by
"folding" the data, that is, adding all the data points based on the
phase at which they were taken to build up a "profile" of the signal
as a function of pulse phase.
> How can I distinguish between deterministic and random
> wander ?
Without some a priori knowledge, it's not easy, though overlapping
short-time FFTs might help. Here I'd produce a grayscale plot of power
versus frequency and time (i.e. which piece the short FFT was applied
to). If you have a nice bright signal, you might see it at the same
frequency all the time, or you might see it drift linearly up or down,
or it might vary sinusoidally in frequency over time, or it might
wander randomly, or it might just always be broad.
If you know exactly how it should be varying, say frequency varying
linearly with time (e.g. Doppler shift from a uniform acceleration) or
sinusoidally, there are special techniques to remove or detect that
modulation. Some of these techniques are done in the time domain;
others by looking at the shape of the peak (if any) in the FFT.
I should say that this is a big enough and complex enough field that
my answers reflect only my experience, which is skewed towards working
with pulsar data, in which the data tends to include very weak but
exactly periodic signals. People used to working with, say, speech
data will have very different ideas on what sort of tools are useful.
More information about the SciPy-User