[SciPy-User] Maximum file size for .npz format?

Christopher Barker Chris.Barker@noaa....
Mon Mar 15 12:50:30 CDT 2010


Nathaniel Smith wrote:
> On Sun, Mar 14, 2010 at 5:24 PM, Christopher Barker
> <Chris.Barker@noaa.gov> wrote:
>> It might be nice to build something on libbzip2 -- It looks like the
>> license is right, it's got good compression qualities, supports 64 bit,
>> and it getting pretty widely used.
> 
> If we're going to pull in a new C library for this purpose, then maybe
> we should just use libhdf5? :-).

except that hdf is one big honking pain in the butt to build.

Robert's right, bzip isn't a great option anyway -- it can do multiple 
files, but it just concatenates them, and doesn't appear to provide an 
index. I used to use afio, which would zip each file first, then put 
them all in an archive -- I liked that approach. We could, of course, 
build something like that with bzip, but it looks like python's zip will 
work for >= 2.6, so no need for something new.

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov


More information about the SciPy-User mailing list