[SciPy-user] Some help on pythonic design of time series data processing application
Sat Jun 13 10:49:22 CDT 2009
> This might be slightly off topic but I'd appreciate any tips or references that anyone may have since I know a lot of you deal with scientific data. Over the last couple of months I have become very familiar with using python/numpy to write lots of small scripts for processing or plotting that I can then string together but I'm having trouble finding descriptions of how to write larger applications and also how to appropriately use classes.
> I'm trying to write an application to semi-automate the process of getting our field data from multiple instruments types into a common format.
> My idea for the workflow (based loosely on stuff we have already sorta implemented using shell scripts) is the following:
> 1) read all files in a specified directory
> 2) parse file names to work out site names and instrument used
> 3) according to the instrument used, load a plugin/call a script that understands how to read in the data from that instrument. The plugin would be aware of what parameters that type of instrument provides and in what units and be able to convert everything to SI units
> 4) append data to a site & parameter specific file that contains all data from that site
> 5) Later I'll be working on a QA/QC application built on top of this.
> I have starting developing a generic plugin class but I'm not sure of how to implement the plugins. In our old shell script based application we wrote an entirely new script for each new instrument and essentially just had a massive IF statement to choose which script to run. This wasn't very scalable and I'd like to be able to add new instruments without having to change the main routines.
> Since I'm not very familiar with python or object oriented programing I was wondering if anyone had any examples or packages they could recommend I look at for ideas. Or if anyone has done something similar please let me know.
I have done something very similar to what you describe for the analysis
of single molecule force spectroscopy data. Have a look at it:
and let me know if I can be of help.
More information about the SciPy-user