[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

David Froger david.froger@gmail....
Tue Mar 20 08:17:12 CDT 2012


Hi,

I think writing a Python script that convert your txt file to one netcdf file,
reading the txt file one line at a time, and then use the netcdf file normally 
would be a good solution!

Best,
David

Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012:
> Dear all,
> 
> I received a file from others which contains ~30 million lines and in size
> of ~500M.
> I try read it with numpy.genfromtxt in ipython interactive mode. Then
> ipython crashed.
> The data contains lat,lon,var1,year, the year ranges from 1001 to 2006.
> Finally I want to write the
> data to netcdf for separate years and feed them into the model. I guess I
> need a better way to do this?
> anyone would be any idea is highly appreciated.
> 
> 
> lon,lat,year,area_burned
> -180.0,65.0,1001,0
> -180.0,65.0,1002,0
> -180.0,65.0,1003,0
> -180.0,65.0,1004,0
> -180.0,65.0,1005,0
> -180.0,65.0,1006,0
> -180.0,65.0,1007,0
> 
> thanks and cheers,
> 
> Chao
> 


More information about the NumPy-Discussion mailing list