[SciPy-User] Maximum file size for .npz format?

Robert Kern robert.kern@gmail....
Fri Mar 12 12:35:48 CST 2010


On Fri, Mar 12, 2010 at 12:27, Anne Archibald <peridot.faceted@gmail.com> wrote:
> On 11 March 2010 10:48, Robert Kern <robert.kern@gmail.com> wrote:
>> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans@gmail.com> wrote:
>>> Hi!
>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>> compains about
>>> BadZipfile: Bad magic number for file header
>>>
>>> I can't open it with the normal zip utility present on the system, but it
>>> could be that it's barfing about files being larger than 2Gb.
>>> Is there some file limit for npzs?
>>
>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>> not yet support the ZIP64 format.
>
> Could it be arranged that an exception is raised when creating a >4GB
> .npz file, so people do not find themselves with unrecoverable data?

If you can arrange it, sure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the SciPy-User mailing list