[SciPy-user] Fast saving/loading of huge matrices
Thu Apr 19 09:23:08 CDT 2007
Gael Varoquaux wrote:
> I have a huge matrix (I don't know how big it is, it hasn't finished
> loading yet, but the ascii file weights 381M). I was wondering what
> format had best speed efficiency for saving/loading huge file. I don't
> mind using a hdf5 even if it is not included in scipy itself.
I think we've found that a simple pickle using protocol 2 works the fastest. At
the time (a year or so ago) this was faster than PyTables for loading the entire
array of about 1GB size. PyTables might be better now, possibly because of the
new numpy support.
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the SciPy-user