[SciPy-User] Maximum file size for .npz format?
Fri Mar 12 12:27:27 CST 2010
On 11 March 2010 10:48, Robert Kern <email@example.com> wrote:
> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <firstname.lastname@example.org> wrote:
>> I need to save a fairly large set of arrays to disk. I have saved it using
>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>> large ;D). When I try to load it using numpy.load, the zipfile module
>> compains about
>> BadZipfile: Bad magic number for file header
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.
Could it be arranged that an exception is raised when creating a >4GB
.npz file, so people do not find themselves with unrecoverable data?
>> Is there anyway I can recover the data (I
>> guess I could try decompressing the file with 7z and extracting the
>> individual npy files?)
> Possibly. However, if the normal zip utility isn't working, 7z
> probably won't, either. Worth a try, though.
If your data is valuable enough — irreplaceable space mission results,
say — some careful spelunking in the code combined with some knowledge
about your data might allow you to semi-manually reconstruct it from
the damaged zip file. This would be a long and painful labour, but
would probably produce a python module that supported zip64 as an
incidental side effect.
> Robert Kern
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> -- Umberto Eco
> SciPy-User mailing list
More information about the SciPy-User