[SciPy-user] High-pass filtering
jl at dmi.dk
Thu May 13 05:51:24 CDT 2004
Thanks for the help Travis,
> SciPy uses Numeric and doesn't require numarray (it can use numarray
> objects as inputs which will be treated as Python sequences and can be
Okay, I did'nt know that (I'm quite new to Python and this is my first use of
SciPy). I thought that numarray was a replacement for Numeric and have
therefore only bothered to learn the former (maybe I should have a look at
Numeric as well).
> What kind of high-pass filter are you looking for. Simple FIR filters
> are quite simple to design and give linear phase. Other IIR filters are
> available. I'll show you how to design a high-pass FIR filter using the
> remez algorithm (you could also look at signal.firwin for windowed
> filter design (but remez is usually a better solution).
I think the FIR filter you have shown how to use is just fine for my purposes
(but I am obviously not an expert on filters). I have therefore described
below what I am going to use it for. If you have any ideas of a better filter
for this application I would be grateful to hear about it.
I'm making an application for quality control (QC) of timeseries of
observations from various meteorological stations. The QC first has several
simple checks. After these checks a check based on optimal interpolation (OI)
is performed. OI is a method to estimate a given observation at a given
station using statistical covariances with timeseries from other stations
(including these timeseries with lagged observations). The OI method will
also give an estimate of the uncertainty of the estimated value. If the real
value is to far away from the estimated value it will be removed or replaced
by the estimated value.
To get good results from the OI it is necessary to remove low frequency
oscillations (timescales > 2-3 months). For example the annual oscillation in
temperature should be removed. If this is not done the covariances used in
the OI will be overshadowed by the annual oscillation and it will be
impossible to estimate fluctuations with high frequency (timescale of
hours-days). The reason for this is that the covariance structure of low and
high frequency oscillations are not the same due to the fact that different
physical processes are responsible for each type of variability.
I hope to release the application as open source when it is complete and has
been tested (it still requires quite some work). It might then be
adapted/expanded to QC of other types of observational timeseries in
completely different areas.
More information about the SciPy-user