[SciPy-user] Reading laaaarge arrays

Travis E. Oliphant oliphant at ee.byu.edu
Wed Sep 22 13:23:40 CDT 2004

Jose Luis Gomez Dans wrote:
> Hi! 
> I have a list of essentially (x,y) data (two columns, 
> in glorious ASCII), that I want to read into a scipy 
> array. The most obvious thing to do is to use 
> io.read_array(), but the files are quite large (up to 
> 0.5Gb), and thus, read_array is very slow.  
> I remember someone mentioning this before, and 
> suggesting a little function written in C, and usable 
> from within scipy to do this efficiently, but I can't 
> find it in the archives or in my mailbox. 
> Can anyone suggest a way to do this? So far, I've 
> been trying to read in a 72Mb file for 17 minutes 
> with io.read_array(), and it's still going!!! 

I've started writing a basic version of scanf for exactly this purpose, 
but I'm not done yet.

For now, my suggestion for such large data is to use weave and sscanf.

io.read_array is quite fancy and not meant for reading simple but very, 
very large files.


More information about the SciPy-user mailing list