[SciPy-user] Fast saving/loading of huge matrices

Robert Kern robert.kern@gmail....
Thu Apr 19 09:23:08 CDT 2007


Gael Varoquaux wrote:
> I have a huge matrix (I don't know how big it is, it hasn't finished
> loading yet, but the ascii file weights 381M). I was wondering what
> format had best speed efficiency for saving/loading huge file. I don't
> mind using a hdf5 even if it is not included in scipy itself.

I think we've found that a simple pickle using protocol 2 works the fastest. At
the time (a year or so ago) this was faster than PyTables for loading the entire
array of about 1GB size. PyTables might be better now, possibly because of the
new numpy support.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


More information about the SciPy-user mailing list