[SciPy-User] Maximum file size for .npz format?

Bruce Southey bsouthey@gmail....
Fri Mar 12 12:45:55 CST 2010


On 03/12/2010 12:35 PM, Robert Kern wrote:
> On Fri, Mar 12, 2010 at 12:27, Anne Archibald<peridot.faceted@gmail.com>  wrote:
>    
>> On 11 March 2010 10:48, Robert Kern<robert.kern@gmail.com>  wrote:
>>      
>>> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans<jgomezdans@gmail.com>  wrote:
>>>        
>>>> Hi!
>>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>>> compains about
>>>> BadZipfile: Bad magic number for file header
>>>>
>>>> I can't open it with the normal zip utility present on the system, but it
>>>> could be that it's barfing about files being larger than 2Gb.
>>>> Is there some file limit for npzs?
>>>>          
>>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>>> not yet support the ZIP64 format.
>>>        
>> Could it be arranged that an exception is raised when creating a>4GB
>> .npz file, so people do not find themselves with unrecoverable data?
>>      
> If you can arrange it, sure.
>
>    
Hi,
Please note that there is the related ticket 991:
http://projects.scipy.org/numpy/ticket/991

Jose, what do Python version are you using and does the ' normal zip 
utility present on the system' actually support zip64?
You should try seeing if 7z sees it.

Bruce


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100312/83d13ef4/attachment.html 


More information about the SciPy-User mailing list