[SciPy-User] scikits.timeseries for many, large, independent and irregular time series

David Nesting david@fastolfe....
Sat Oct 22 12:22:43 CDT 2011


I am interested in using scikits.timeseries for a project.  I expect to use
it with hundreds or thousands of sensors recording points of data.  I need
to perform computations on the resulting timeseries as data points arrive.
 Most of these computations will be on recent data, but some will need to
look backward much farther.  The reporting rate for each incoming data point
will vary from a second (perhaps less) to a minute or longer, and the time
at which the points arrive will not be perfectly regular (some sensors will
only report changes).

1. I've seen posts discussing converting irregular timeseries to "proper"
regularly spaced TimeSeries data.  But since this loses data fidelity, and
not all values are meant to be interpolated, I want to keep the original
data points, so it seems like I would need to store (and persist) this data
using data structures and methods of my own devising, and only use
TimeSeries objects when I want to do computations on them (and at that point
I can make them regular and fill in or interpolate empty values as needed).
 Does that sound right?

2. Some computations could involve very large TimeSeries objects.  The
original data points may not even be in memory and would need to be fetched
from a database.  It seems like I cannot avoid putting these data points in
memory in order to perform the computations, though.  Is there any support
(or suggested techniques) for doing streaming operations on TimeSeries?  In
other words, if I want to get an average (across time) of the sum (across
space) of 10 000 different TimeSeries, I should be able to compute that
incrementally without having to hold any of those TimeSeries in memory.
 And, ideally, if some of those data points need to be fetched from a
database (or a back-end server), it should be possible to do that fairly
transparently.

Any advice here?  I appreciate your help.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20111022/3da15923/attachment-0001.html 


More information about the SciPy-User mailing list