[Numpy-discussion] NPZ format

Francesc Alted faltet@pytables....
Fri Oct 1 02:13:40 CDT 2010


A Thursday 30 September 2010 18:20:16 Robert Kern escrigué:
> On Wed, Sep 29, 2010 at 03:17, Francesc Alted <faltet@pytables.org> 
wrote:
> > Hi,
> > 
> > I'm going to give a seminar about serialization, and I'd like to
> > describe the .npy format.  I noticed that there is a variant of it
> > called .npz that can pack several arrays in one single file.
> > 
> > However, .npz does not use compression at all and I'm wondering
> > what's the reason.  I suppose that this is because you don't want
> > to loose the possibility to memmap saved arrays, but can someone
> > confirm this?
> 
> While I suspect it's possible, I'm certain we don't have any code
> that actually does it. Most likely the author assumed that it would
> be faster (or tested it to be faster with their CPU/hard disk
> configuration) to not compress.

Thanks, that's good to know.  And yes, I'd say that compressing with zip 
(zlib) would reduce performance for doing I/O, but most probably 
decompressing from disk media would represent an improvement in terms of 
time.  At any rate, adding compression capability to .npy should be just 
one parameter away, so perhaps is a good idea adding it.

-- 
Francesc Alted


More information about the NumPy-Discussion mailing list