[SciPy-User] Maximum file size for .npz format?

Robert Kern robert.kern@gmail....
Fri Mar 12 12:35:48 CST 2010

On Fri, Mar 12, 2010 at 12:27, Anne Archibald <peridot.faceted@gmail.com> wrote:
> On 11 March 2010 10:48, Robert Kern <robert.kern@gmail.com> wrote:
>> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans@gmail.com> wrote:
>>> Hi!
>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>> compains about
>>> BadZipfile: Bad magic number for file header
>>> I can't open it with the normal zip utility present on the system, but it
>>> could be that it's barfing about files being larger than 2Gb.
>>> Is there some file limit for npzs?
>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>> not yet support the ZIP64 format.
> Could it be arranged that an exception is raised when creating a >4GB
> .npz file, so people do not find themselves with unrecoverable data?

If you can arrange it, sure.

Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

More information about the SciPy-User mailing list