[Numpy-discussion] fast numpy i/o

Christopher Barker Chris.Barker@noaa....
Tue Jun 21 13:35:01 CDT 2011

Robert Kern wrote:
> https://raw.github.com/numpy/numpy/master/doc/neps/npy-format.txt

Just a note. From that doc:

     HDF5 is a complicated format that more or less implements
     a hierarchical filesystem-in-a-file.  This fact makes satisfying
     some of the Requirements difficult.  To the author's knowledge, as
     of this writing, there is no application or library that reads or
     writes even a subset of HDF5 files that does not use the canonical
     libhdf5 implementation.

I'm pretty sure that the NetcdfJava libs, developed by Unidata, use 
their own home-grown code. netcdf4 is built on HDF5, so that qualifies 
as "a library that reads or writes a subset of HDF5 files". Perhaps 
there are lessons to be learned there. (too bad it's Java)

     Furthermore, by
     providing the first non-libhdf5 implementation of HDF5, we would
     be able to encourage more adoption of simple HDF5 in applications
     where it was previously infeasible because of the size of the

I suppose this point is still true -- a C lib that supported a subset of 
hdf would be nice.

That being said, I like the simplicity of the .npy format, and I don't 
know that anyone wants to take any of this on anyway.


Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception


More information about the NumPy-Discussion mailing list