[Numpy-discussion] [SciPy-User] recommendation for saving data
Mon Aug 1 12:17:33 CDT 2011
In astronomy we tend to use FITS, which is well-supported by pyfits,
but a little limited. Some new instruments are beginning to use HDF5.
All these generic formats allow very general data storage, so you will
need to come up with a standrdized way to represent your own data.
Used well, these formats can be self-describing enough that generic
tools can be very useful (e.g. display images, build histograms) but
it takes some thought when designing files.
On 8/1/11, Christopher Barker <Chris.Barker@noaa.gov> wrote:
> On 7/31/11 5:48 AM, Brian Blais wrote:
>> I was wondering if there are any recommendations for formats for saving
>> scientific data.
> every field has it's own standards -- I'd try to find one that is likely
> to be used by folks that may care about your results.
> For Oceanographic and Atmospheric modeling data, netcdf is a good
> option. I like the NetCDF4 python lib:
> (there are others)
> For broader use, and a bit more flexibility, HDF is a good option. There
> are at least two ways to use it with numpy:
> PyTables: http://www.pytables.org
> (Nice higher-level interface)
> (a more raw HDF5 wrapper)
> There is also the npz format, built in to numpy, if you are happy with
> requiring python to read the data.
> I am running a simulation, which has many somewhat-indepedent parts
> which have their own internal state and parameters. I've been using
> pickle (gzipped) to save the entire object (which contains subobjects,
> etc...), but it is getting too unwieldy and I think it is time to look
> for a more robust solution. Ideally I'd like to have something where I
> can call a save method on the simulation object, and it will call the
> save methods on all the children, on down the line all saving into one
> file. It'd also be nice if it were cross-platform, and I could depend
> on the files being readable into the future for a while.
>> Are there any good standards for this? What do you use for saving
>> scientific data?
>> thank you,
>> Brian Blais
> Christopher Barker, Ph.D.
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
> NumPy-Discussion mailing list
Sent from my mobile device
More information about the NumPy-Discussion