[Numpy-discussion] ANN: HDF5 for Python 1.1

Andrew Collette h5py@alfven....
Tue Feb 10 03:06:10 CST 2009


Hi Stephen,

There are no immediate plans to support LZO in h5py, and in fact I'm
starting to regret including any fast compressor at all as I'm now
responsible for maintaining it. :) The reason for the dichotomy is
that LZO is released under the GPL, which is incompatible with h5py's
license.  The LZF license lets me embed it in the distribution, which
(1) frees me from an external dependency, and (2) means you can always
open h5py-created files with h5py, regardless of the platform or build
options.

Besides which, I figured people who used PyTables LZO were happy using
PyTables. :)

Andrew Collette

On Mon, Feb 9, 2009 at 10:30 PM, Stephen Simmons <mail@stevesimmons.com> wrote:
> Hi Andrew,
>
> Do you have any plans to support LZO compression in h5py?
>
> I have lots of LZO-compressed datasets created with PyTables.
> There's a real barrier to using both h5py and PyTables if the fast
> decompressor options are just LZF on h5py and LZO on PyTables.
>
> Many thanks
> Stephen
>
>
> Andrew Collette wrote:
>> =====================================
>> Announcing HDF5 for Python (h5py) 1.1
>> =====================================
>>
>> What is h5py?
>> -------------
>>
>> HDF5 for Python (h5py) is a general-purpose Python interface to the
>> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
>> mature scientific software library designed for the fast, flexible
>> storage of enormous amounts of data.
>>
>> >From a Python programmer's perspective, HDF5 provides a robust way to
>> store data, organized by name in a tree-like fashion.  You can create
>> datasets (arrays on disk) hundreds of gigabytes in size, and perform
>> random-access I/O on desired sections.  Datasets are organized in a
>> filesystem-like hierarchy using containers called "groups", and
>> accesed using the tradional POSIX /path/to/resource syntax.
>>
>> In addition to providing interoperability with existing HDF5 datasets
>> and platforms, h5py is a convienient way to store and retrieve
>> arbitrary NumPy data and metadata.
>>
>>
>> New features in 1.1
>> -------------------
>>
>>   - A new compression filter based on the LZF library, which provides
>>     transparent compression many times faster than the standard HDF5
>>     GZIP filter.
>>
>>   - Efficient broadcasting using HDF5 hyperslab selections; for example,
>>     you can write to a (2000 x 100 x 50) selection from a (100 x 50)
>>     source array.
>>
>>   - Now supports the NumPy boolean type
>>
>>   - Auto-completion for IPython 0.9.X (contributed by Darren Dale)
>>
>>   - Installable via easy_install
>>
>>
>> Standard features
>> -----------------
>>
>>   - Supports storage of NumPy data of the following types:
>>
>>     * Integer/Unsigned Integer
>>     * Float/Double
>>     * Complex/Double Complex
>>     * Compound ("recarray")
>>     * Strings
>>     * Boolean
>>     * Array (as members of a compound type only)
>>     * Void
>>
>>   - Random access to datasets using the standard NumPy slicing syntax,
>>     including fancy indexing and point-based selection
>>
>>   - Transparent compression of datasets using GZIP, LZF or SZIP,
>>     and error-detection using Fletcher32
>>
>>   - "Pythonic" interface supporting dictionary and NumPy-array metaphors
>>     for the high-level HDF5 abstrations like groups and datasets
>>
>>   - A comprehensive, object-oriented wrapping of the HDF5 low-level C API
>>     via Cython, in addition to the NumPy-like high-level interface.
>>
>>   - Supports many new features of HDF5 1.8, including recursive iteration
>>     over entire files and in-library copy operations on the file tree
>>
>>   - Thread-safe
>>
>>
>> Where to get it
>> ---------------
>>
>> * Main website, documentation:  http://h5py.alfven.org
>>
>> * Downloads, bug tracker:       http://h5py.googlecode.com
>>
>>
>> Requires
>> --------
>>
>> * Linux, Mac OS-X or Windows
>>
>> * Python 2.5 (Windows), Python 2.5 or 2.6 (Linux/Mac OS-X)
>>
>> * NumPy 1.0.3 or later
>>
>> * HDF5 1.6.5 or later (including 1.8); HDF5 is included with
>>   the Windows version.
>>
>>
>> Thanks
>> ------
>>
>> Thanks to D. Dale, E. Lawrence and other for their continued support
>> and comments.  Also thanks to the Francesc Alted and the PyTables project,
>> for inspiration and generously providing their code to the community. Thanks
>> to everyone at the HDF Group for creating such a useful piece of software.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion@scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


More information about the Numpy-discussion mailing list