[SciPy-user] read/write compressed files

Dominik Szczerba domi@vision.ee.ethz...
Thu Jun 21 06:45:13 CDT 2007


There is also another thing, namely bz2 uses --best per default while
gzip uses -6. The whole thing is of course strongly data-dependent.
- Dominik

Francesc Altet wrote:
> El dj 21 de 06 del 2007 a les 12:57 +0200, en/na Dominik Szczerba va
> escriure:
>> Hi,
>>
>> I meant bz2 over zlib due to higher compression, if slower performance.
>> This common belief was usually parallel to my experience. However, a
>> simple test below made with fresh morning data clearly undermines this
>> thinking:
>>
>>
>>
>>> du -hsc test9*.dat
>> 428M    total
>>
>>> time gzip test9*.dat
>> real    0m31.663s
>> user    0m28.946s
>> sys     0m1.612s
>>
>>> du -hsc test9*.dat.gz
>> 215M    total
>>
>>> time gunzip test9*.dat.gz
>> real    0m7.447s
>> user    0m6.036s
>> sys     0m1.264s
>>
>>> time bzip2 test9*.dat
>> real    2m1.696s
>> user    1m54.527s
>> sys     0m4.008s
>>
>>> du -hsc test9*.dat.bz2
>> 219M    total
>>
>>> time bunzip2 test9*.dat.bz2
>> real    0m43.252s
>> user    0m39.926s
>> sys     0m2.792s
>>
>>
>> I am surprised, as I well remember cases where I could gain 20%.
> 
> Yeah, there should be cases where bzip2 is clearly better than zlib and
> one of these could be images.  My teammate Ivan has come with this
> example:
> 
> -rw------- 1 ivan ivan 733373 2007-06-21 13:02 lena1.tif.gz
> -rw------- 1 ivan ivan 584478 2007-06-21 13:02 lena2.tif.bz2
> 
> (you should already know where the source is: www.lenna.org )
> 
> But when it comes to general binary data for scientific uses, the
> compression advantages of bzip2 over zlib are less clear.
> 
>>  But
>> indeed, given the much slower performance, you have me convinced to use
>> zlib over bz2.
>>
>> thanks for forcing me to do this test,
> 
> You are welcome ;)
> 

-- 
Dominik Szczerba, Ph.D.
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi


More information about the SciPy-user mailing list