[SciPy-user] High-pass filtering

Jesper Larsen jl at dmi.dk
Thu May 13 05:51:24 CDT 2004


Thanks for the help Travis,

> SciPy uses Numeric and doesn't require numarray (it can use numarray
> objects as inputs which will be treated as Python sequences and can be
> slow).

Okay, I did'nt know that (I'm quite new to Python and this is my first use of 
SciPy). I thought that numarray was a replacement for Numeric and have 
therefore only bothered to learn the former (maybe I should have a look at 
Numeric as well).

> What kind of high-pass filter are you looking for.  Simple FIR filters
> are quite simple to design and give linear phase.  Other IIR filters are
> available.  I'll show you how to design a high-pass FIR filter using the
> remez algorithm (you could also look at signal.firwin for windowed
> filter design (but remez is usually a better solution).
...

I think the FIR filter you have shown how to use is just fine for my purposes 
(but I am obviously not an expert on filters). I have therefore described 
below what I am going to use it for. If you have any ideas of a better filter 
for this application I would be grateful to hear about it.

I'm making an application for quality control (QC) of timeseries of 
observations from various meteorological stations. The QC first has several 
simple checks. After these checks a check based on optimal interpolation (OI) 
is performed. OI is a method to estimate a given observation at a given 
station using statistical covariances with timeseries from other stations 
(including these timeseries with lagged observations). The OI method will 
also give an estimate of the uncertainty of the estimated value. If the real 
value is to far away from the estimated value it will be removed or replaced 
by the estimated value.

To get good results from the OI it is necessary to remove low frequency 
oscillations (timescales > 2-3 months). For example the annual oscillation in 
temperature should be removed. If this is not done the covariances used in 
the OI will be overshadowed by the annual oscillation and it will be 
impossible to estimate fluctuations with high frequency (timescale of 
hours-days). The reason for this is that the covariance structure of low and 
high frequency oscillations are not the same due to the fact that different 
physical processes are responsible for each type of variability.

I hope to release the application as open source when it is complete and has 
been tested (it still requires quite some work). It might then be 
adapted/expanded to QC of other types of observational timeseries in 
completely different areas.

Kind regards,
Jesper



More information about the SciPy-user mailing list