[SciPy-user] What is fastest load/save matrix methods?
cournape at atr.jp
Mon Dec 19 19:49:14 CST 2005
On Mon, 2005-12-19 at 11:01 +0100, Arnd Baecker wrote:
> On Mon, 19 Dec 2005, Hugo Gamboa wrote:
> > Hi there,
> > I need to work with large matrixes that come in ascii format.
> > I have the functions to build the matrix from the ascii file but it
> > takes too long.
> > I can spend some time convert ingthis ascii files to some other format
> > so that the re-readeing be fastened.
> > What is the fastest (binary) format readly available in scipy to
> > load/save matrixes?
> I am not sure if it is *the* fastest way, but have you had a look
> at scipy.io?
Did you take a look at pytables ? It is a python library built around
the hdf5 file format
HDF5 was created to address the data management needs of scientists and
engineers working in high performance, data intensive computing
environments. As a result, the HDF5 library and format emphasize storage
and I/O efficiency. For instance, the HDF5 format can accommodate data
in a variety of ways, such as compressed or chunked. And the library is
tuned and adapted to read and write data efficiently on parallel
pytables uses hdf5 file format as a storage model, with a high level
abstraction when you want to save a lot of data in one file.
The lastest version can read a lot of native hdf5 files, not just
pytables files, so a file created through the hdf5 C library should be
usable under python with pytables.
More information about the SciPy-user