[Numpy-discussion] How to read data from text files fast?

[Chris Barker]

Hi all,

I'm looking for a way to read data from ascii text files quickly. I've 
found that using the standard python idioms like:

data = array((M,N),Float)
for in range(N):
     data.append(map(float,file.readline().split()))

Can be pretty slow. What I'd like is something like Matlab's fscanf:

data = fscanf(file, "%g", [M,N] )

I may have the syntax a little wrong, but the gist is there. What Matlab 
does keep recycling the format string until the desired number of 
elements have been read.

It is quite flexible, and ends up being pretty fast.

Has anyone written something like this for Numeric (or numarray, but I'd 
prefer Numeric at this point) ?

I was surprised not to find something like this in SciPy, maybe I didn't 
look hard enough.

If no one has done this, I guess I'll get started on it....

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov