[Numpy-discussion] reading gzip compressed files using numpy.fromfile

[Peter Schmidtke]

> Date: Wed, 28 Oct 2009 20:31:43 +0100
> From: Peter Schmidtke <pschmidtke@mmb.pcb.ub.es>
> Subject: [Numpy-discussion] reading gzip compressed files using
> 	numpy.fromfile
> To: numpy-discussion@scipy.org
> Message-ID: <fc345224bfa26132e9474287e32e083b@mmb.pcb.ub.es>
> Content-Type: text/plain; charset="UTF-8"
> 
> Dear Numpy Mailing List Readers,
> 
> I have a quite simple problem, for what I did not find a solution for
now. 
> I have a gzipped file lying around that has some numbers stored in it and
I
> want to read them into a numpy array as fast as possible but only a bunch
> of data at a time. 
> So I would like to use numpys fromfile funtion. 
> 
> For now I have somehow the following code :
> 
> 
> 
>         f=gzip.open( "myfile.gz", "r" )
> xyz=npy.fromfile(f,dtype="float32",count=400) 
> 
> 
> So I would read 400 entries from the file, keep it open, process my data,
> come back and read the next 400 entries. If I do this, numpy is
complaining
> that the file handle f is not a normal file handle :
> OError: first argument must be an open file
> 
> but in fact it is a zlib file handle. But gzip gives access to the normal
> filehandle through f.fileobj.
> 
> So I tried  xyz=npy.fromfile(f.fileobj,dtype="float32",count=400)
> 
> But there I get just meaningless values (not the actual data) and when I
> specify the sep=" " argument for npy.fromfile I get just .1 and nothing
> else. 
> 
> Can you tell me why and how to fix this problem? I know that I could read
> everything to memory, but these files are rather big, so I simply have to
> avoid this.
> 
> Thanks in advance.
> 
> 
> -- 
> 
> Peter Schmidtke
> 
> ----------------------
> PhD Student at the Molecular Modeling and Bioinformatics Group
> Dep. Physical Chemistry
> Faculty of Pharmacy
> University of Barcelona
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 28 Oct 2009 14:33:11 -0500
> From: Robert Kern <robert.kern@gmail.com>
> Subject: Re: [Numpy-discussion] reading gzip compressed files using
> 	numpy.fromfile
> To: Discussion of Numerical Python <numpy-discussion@scipy.org>
> Message-ID:
> 	<3d375d730910281233r5cadd0fcubea14676a3a978f1@mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
> 
> On Wed, Oct 28, 2009 at 14:31, Peter Schmidtke <pschmidtke@mmb.pcb.ub.es>
> wrote:
>> Dear Numpy Mailing List Readers,
>>
>> I have a quite simple problem, for what I did not find a solution for
>> now.
>> I have a gzipped file lying around that has some numbers stored in it
and
>> I
>> want to read them into a numpy array as fast as possible but only a
bunch
>> of data at a time.
>> So I would like to use numpys fromfile funtion.
>>
>> For now I have somehow the following code :
>>
>>
>>
>> ? ? ? ?f=gzip.open( "myfile.gz", "r" )
>> xyz=npy.fromfile(f,dtype="float32",count=400)
>>
>>
>> So I would read 400 entries from the file, keep it open, process my
data,
>> come back and read the next 400 entries. If I do this, numpy is
>> complaining
>> that the file handle f is not a normal file handle :
>> OError: first argument must be an open file
>>
>> but in fact it is a zlib file handle. But gzip gives access to the
normal
>> filehandle through f.fileobj.
> 
> np.fromfile() requires a true file object, not just a file-like
> object. np.fromfile() works by grabbing the FILE* pointer underneath
> and using C system calls to read the data, not by calling the .read()
> method.
> 
>> So I tried ?xyz=npy.fromfile(f.fileobj,dtype="float32",count=400)
>>
>> But there I get just meaningless values (not the actual data) and when I
>> specify the sep=" " argument for npy.fromfile I get just .1 and nothing
>> else.
> 
> This is reading the compressed data, not the data that you want.
> 
>> Can you tell me why and how to fix this problem? I know that I could
read
>> everything to memory, but these files are rather big, so I simply have
to
>> avoid this.
> 
> Read in reasonably-sized chunks of bytes at a time, and use
> np.fromstring() to create arrays from them.
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 28 Oct 2009 13:26:41 -0700
> From: Christopher Barker <Chris.Barker@noaa.gov>
> Subject: Re: [Numpy-discussion] reading gzip compressed files	using
> 	numpy.fromfile
> To: Discussion of Numerical Python <numpy-discussion@scipy.org>
> Message-ID: <4AE8A901.3060403@noaa.gov>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> Robert Kern wrote:
>>>        f=gzip.open( "myfile.gz", "r" )
>>> xyz=npy.fromfile(f,dtype="float32",count=400)
> 
>> Read in reasonably-sized chunks of bytes at a time, and use
>> np.fromstring() to create arrays from them.
> 
> Something like:
> 
> count = 400
> xyz = np.fromstring(f.read(count*4), dtype=np.float32)
> 
> should work (untested...)
> 
> -Chris
> 
> 
> 
> 
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker@noaa.gov
> 
> 

Thanks Robert and Chris...indeed I managed to read it quite fast this way.

++


Peter Schmidtke

----------------------
PhD Student at the Molecular Modeling and Bioinformatics Group
Dep. Physical Chemistry
Faculty of Pharmacy
University of Barcelona