[SciPy-user] Reading laaaarge arrays
Travis E. Oliphant
oliphant at ee.byu.edu
Wed Sep 22 13:23:40 CDT 2004
Jose Luis Gomez Dans wrote:
> I have a list of essentially (x,y) data (two columns,
> in glorious ASCII), that I want to read into a scipy
> array. The most obvious thing to do is to use
> io.read_array(), but the files are quite large (up to
> 0.5Gb), and thus, read_array is very slow.
> I remember someone mentioning this before, and
> suggesting a little function written in C, and usable
> from within scipy to do this efficiently, but I can't
> find it in the archives or in my mailbox.
> Can anyone suggest a way to do this? So far, I've
> been trying to read in a 72Mb file for 17 minutes
> with io.read_array(), and it's still going!!!
I've started writing a basic version of scanf for exactly this purpose,
but I'm not done yet.
For now, my suggestion for such large data is to use weave and sscanf.
io.read_array is quite fancy and not meant for reading simple but very,
very large files.
More information about the SciPy-user