[Numpy-discussion] Reading a big netcdf file

[Christopher Barker]

On 8/3/11 9:30 AM, Kiko wrote:
> I'm trying to read a big netcdf file (445 Mb) using netcdf4-python.

I've never noticed that netCDF4 was particularly slow for reading 
(writing can be pretty slow some times). How slow is slow?

> The data are described as:

please post the results of:

ncdump -h the_file_name.nc

So we can see if there is anything odd in the structure (though I don't 
know what it might be)

Post your code (in the simnd pplest form you can).

and post your timings and machine type

Is the file netcdf4 or 3 format? (the python lib will read either)

As a reference, reading that much data in from a raw file into a numpy 
array takes 2.57 on my machine (a rather old Mac, but disks haven't 
gotten much  faster). YOu can test that like this:

a = np.zeros((21601, 10801), dtype=np.uint16)

a.tofile('temp.npa')

del a

timeit a = np.fromfile('temp.npa', dtype=np.uint16)

(using ipython's timeit)

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov