[SciPy-user] What is fastest load/save matrix methods?

Cournapeau David cournape at atr.jp
Mon Dec 19 19:49:14 CST 2005


On Mon, 2005-12-19 at 11:01 +0100, Arnd Baecker wrote:
> On Mon, 19 Dec 2005, Hugo Gamboa wrote:
> 
> > Hi there,
> >
> > I need to work with large matrixes that come in ascii format.
> >
> > I have the functions to build the matrix from the ascii file but it
> > takes too long.
> > I can spend some time convert ingthis ascii files to some other format
> > so that the re-readeing be fastened.
> >
> > What is the fastest (binary) format readly available in scipy to
> > load/save matrixes?
> 
> I am not sure if it is *the* fastest way, but have you had a look
> at scipy.io?
Did you take a look at pytables ? It is a python library built around
the hdf5 file format

"""
HDF5 was created to address the data management needs of scientists and
engineers working in high performance, data intensive computing
environments. As a result, the HDF5 library and format emphasize storage
and I/O efficiency. For instance, the HDF5 format can accommodate data
in a variety of ways, such as compressed or chunked. And the library is
tuned and adapted to read and write data efficiently on parallel
computing systems.
"""

pytables uses hdf5 file format as a storage model, with a high level
abstraction when you want to save a lot of data in one file.

http://pytables.sourceforge.net/html/WelcomePage.html

The lastest version can read a lot of native hdf5 files, not just
pytables files, so a file created through the hdf5 C library should be
usable under python with pytables.

David



More information about the SciPy-user mailing list