[Numpy-discussion] [SciPy-User] recommendation for saving data

Anne Archibald aarchiba@physics.mcgill...
Mon Aug 1 12:17:33 CDT 2011

In astronomy we tend to use FITS, which is well-supported by pyfits,
but a little limited. Some new instruments are beginning to use HDF5.

All these generic formats allow very general data storage, so you will
need to come up with a standrdized way to represent your own data.
Used well, these formats can be self-describing enough that generic
tools can be very useful (e.g. display images, build histograms) but
it takes some thought when designing files.


On 8/1/11, Christopher Barker <Chris.Barker@noaa.gov> wrote:
> On 7/31/11 5:48 AM, Brian Blais wrote:
>> I was wondering if there are any recommendations for formats for saving
>> scientific data.
> every field has it's own standards -- I'd try to find one that is likely
> to be used by folks that may care about your results.
> For Oceanographic and Atmospheric modeling data, netcdf is a good
> option. I like the NetCDF4 python lib:
> http://code.google.com/p/netcdf4-python/
> (there are others)
> For broader use, and a bit more flexibility, HDF is a good option. There
> are at least two ways to use it with numpy:
> PyTables: http://www.pytables.org
> (Nice higher-level interface)
> hf5py:
> http://alfven.org/wp/hdf5-for-python/
> (a more raw HDF5 wrapper)
> There is also the npz format, built in to numpy, if you are happy with
> requiring python to read the data.
> -Chris
>   I am running a simulation, which has many somewhat-indepedent parts
> which have their own internal state and parameters.  I've been using
> pickle (gzipped) to save the entire object (which contains subobjects,
> etc...), but it is getting too unwieldy and I think it is time to look
> for a more robust solution.  Ideally I'd like to have something where I
> can call a save method on the simulation object, and it will call the
> save methods on all the children, on down the line all saving into one
> file.  It'd also be nice if it were cross-platform, and I could depend
> on the files being readable into the future for a while.
>> Are there any good standards for this?  What do you use for saving
>> scientific data?
>> 		thank you,
>> 			Brian Blais
> --
> Christopher Barker, Ph.D.
> Oceanographer
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> Chris.Barker@noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Sent from my mobile device

More information about the NumPy-Discussion mailing list