[SciPy-User] HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??
Mon Nov 1 17:13:56 CDT 2010
Ack -- I didn't mean to set off a big back and forth! I'd only wanted
to convey that h5py and pytables seem to serve different purposes: one
a "simple and thin" pythonic wrapper around the official hdf5
libraries, and the other seeking to provide some value-add on top of
I guess I got something of the rationale for using h5py over pytables
wrong -- but there is some rationale, right? What is that?
On Nov 1, 2010, at 6:02 PM, Francesc Alted wrote:
> A Monday 01 November 2010 21:19:02 Robert Kern escrigué:
>> On Mon, Nov 1, 2010 at 15:01, Dav Clark <firstname.lastname@example.org> wrote:
>>> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>>>> Where pytables tries to present its own interface, h5py just gives
>>>> you the hdf5 file. This means that pytables can do a lot of neat
>>>> things (like the indexed searching), but it also means that (at
>>>> least last I checked) pytables isn't the best tool for reading in
>>>> hdf5 files not created by pytables -- for that, you'd want h5py.
>>> Every time I've had an issue with pytables reading a non-pytables
>>> created file, I've submitted a bug and it got fixed usually in a
>>> few days. At the time, I was using HDF5 as a transfer layer
>>> between matlab's rudimentary hdf5 support and python w/ pytables.
>>> (Thanks Francesc!)
>> I just wanted to add that in my experience, you can read just about
>> any HDF5 file with PyTables except for a few with some more exotic
> Let me chime in just to try to clarify couple of things. First, both
> PyTables and h5py can read most of the HDF5 files out there, but
> none of
> them has *complete* support for HDF5 files (implementing complete
> support for the whole HDF5 standard is really a tough task). In
> addition, the last time that I checked this (about one year ago, so
> things might have changed since then), PyTables can read (and create)
> HDF5 files that h5py cannot; and the contrary is true too.
>> If you absolutely need to write an HDF5 file according to a
>> strict standard without any extra bits, you may need h5py. However,
>> many other readers of your standard probably won't care about the
>> extra bits PyTables includes.
> I suppose that the 'extra bits' you are referring to are the HDF5
> attributes that complement HDF5 nodes as metainfo. Let me say that
> of these attributes are not PyTables-specific, but those used in the
> high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).
> as I said many times, if these attributes are causing some trouble to
> the user (they should not), you can always disable its creation by
> setting the PYTABLES_SYS_ATTRS parameter to false during the opening
> a file (or, if you like this to be permanent, in the
> tables/parameters.py). For more info about this, see:
>> You just have to be a little bit
>> careful to make sure that you aren't relying on any PyTables
>> features, like Python-pickled attributes.
> PyTables only uses pickle when trying to save attributes that are not
> supported by HDF5 (with the exception of unicode strings that should
> implemented soon in PyTables). For example, if you try to save a list
> as an attribute:
> node.attrs.my_attr = [1,2,[3,4]]
> as such a list cannot be represented by HDF5 natively, PyTables
> to pickle it and save it. During retrieval, the pickle is
> detected and unpickled before being returned to the user. Of course,
> you will not be able to read such attributes with a non-Python
> application. And, although I consider this like a feature, I can
> understand that this might be considered as a bug by others (but I
> to say that very few PyTables users, if any at all, has ever
> about this 'feature'/'bug').
> Hope this helps clarifying some points,
> Francesc Alted
> SciPy-User mailing list
More information about the SciPy-User