[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

Chao YUE chaoyuejoy@gmail....
Tue Mar 20 08:40:33 CDT 2012


I would be in agree. thanks!
I use gawk to separate the file into many files by year, then it would be
easier to handle.
anyway, it's not a good practice to produce such huge line txt files....

Chao

2012/3/20 David Froger <david.froger@gmail.com>

> Hi,
>
> I think writing a Python script that convert your txt file to one netcdf
> file,
> reading the txt file one line at a time, and then use the netcdf file
> normally
> would be a good solution!
>
> Best,
> David
>
> Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012:
> > Dear all,
> >
> > I received a file from others which contains ~30 million lines and in
> size
> > of ~500M.
> > I try read it with numpy.genfromtxt in ipython interactive mode. Then
> > ipython crashed.
> > The data contains lat,lon,var1,year, the year ranges from 1001 to 2006.
> > Finally I want to write the
> > data to netcdf for separate years and feed them into the model. I guess I
> > need a better way to do this?
> > anyone would be any idea is highly appreciated.
> >
> >
> > lon,lat,year,area_burned
> > -180.0,65.0,1001,0
> > -180.0,65.0,1002,0
> > -180.0,65.0,1003,0
> > -180.0,65.0,1004,0
> > -180.0,65.0,1005,0
> > -180.0,65.0,1006,0
> > -180.0,65.0,1007,0
> >
> > thanks and cheers,
> >
> > Chao
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20120320/e3ebd7ee/attachment-0001.html 


More information about the NumPy-Discussion mailing list