[IPython-User] ipython crash when trying to read 500M txt data in interactive mode

[Giovanni Luca Ciampaglia]

Hi, from your fragment it seems that all your variables are integers. 
Would it make sense to pass dtype=np.int32 to save memory?

Does genfromtxt crash when you call it in non-interactive mode (i.e. 
from inside a script)?

Anyway once you manage to load as an array I would recommend to save the 
data as a binary array file (numpy.save), because loading is 
dramatically faster with that format.

Cheers

G

On 03/20/2012 01:32 PM, Chao YUE wrote:
> Dear all,
>
> I received a file from others which contains ~30 million lines and in
> size of ~500M.
> I try read it with numpy.genfromtxt in ipython interactive mode. Then
> ipython crashed.
> The data contains lat,lon,var1,year, the year ranges from 1001 to 2006.
> Finally I want to write the
> data to netcdf for separate years and feed them into the model. I guess
> I need a better way to do this?
> anyone would be any idea is highly appreciated.
>
>
> lon,lat,year,area_burned
> -180.0,65.0,1001,0
> -180.0,65.0,1002,0
> -180.0,65.0,1003,0
> -180.0,65.0,1004,0
> -180.0,65.0,1005,0
> -180.0,65.0,1006,0
> -180.0,65.0,1007,0
>
> thanks and cheers,
>
> Chao
> --
> ***********************************************************************************
> Chao YUE
> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> UMR 1572 CEA-CNRS-UVSQ
> Batiment 712 - Pe 119
> 91191 GIF Sur YVETTE Cedex
> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
> ************************************************************************************
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user


-- 
Giovanni Luca Ciampaglia

Ph.D. Candidate
Faculty of Informatics
University of Lugano
Web: http://www.inf.usi.ch/phd/ciampaglia/

Bertastraße 36 ∙ 8003 Zürich ∙ Switzerland

Mobile: +41 79 718 8157