[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
Tue Mar 20 08:17:12 CDT 2012
I think writing a Python script that convert your txt file to one netcdf file,
reading the txt file one line at a time, and then use the netcdf file normally
would be a good solution!
Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012:
> Dear all,
> I received a file from others which contains ~30 million lines and in size
> of ~500M.
> I try read it with numpy.genfromtxt in ipython interactive mode. Then
> ipython crashed.
> The data contains lat,lon,var1,year, the year ranges from 1001 to 2006.
> Finally I want to write the
> data to netcdf for separate years and feed them into the model. I guess I
> need a better way to do this?
> anyone would be any idea is highly appreciated.
> thanks and cheers,
More information about the NumPy-Discussion