[SciPy-User] Maximum file size for .npz format?
Fri Mar 12 11:29:58 CST 2010
On Fri, Mar 12, 2010 at 11:22 AM, Paul Anton Letnes <
> On 11. mars 2010, at 23.50, Lafras Uys wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >>> I need to save a fairly large set of arrays to disk. I have saved it
> >>> numpy.savez, and the resulting file is around 11Gb (yes, I did say
> >>> large ;D). When I try to load it using numpy.load, the zipfile module
> >>> compains about
> >>> BadZipfile: Bad magic number for file header
> >>> I can't open it with the normal zip utility present on the system, but
> >>> could be that it's barfing about files being larger than 2Gb.
> >>> Is there some file limit for npzs?
> >> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> >> not yet support the ZIP64 format.
> >>> Is there anyway I can recover the data (I
> >>> guess I could try decompressing the file with 7z and extracting the
> >>> individual npy files?)
> >> Possibly. However, if the normal zip utility isn't working, 7z
> >> probably won't, either. Worth a try, though.
> > I've had similar problems, my solution was to move to HDF5. There are
> > two options for accessing and working with HDF files from python: h5py
> > (http://code.google.com/p/h5py/) and pytables
> > (http://www.pytables.org/). Both packages have built in numpy support.
> > Regards,
> > Lafras
> I've experienced similar issues too, but I moved to NetCDF. The only
> disadvantage was that I did not find any python modules that work well _and_
> support numpy. Hence, I am considering moving to HDF5. Which python module
> would people here recommend? (Or, alternatively, did I miss a great netCDF
> python module that someone could tell me about?)
> SciPy-User mailing list
There is http://code.google.com/p/netcdf4-python/
I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the SciPy-User