[SciPy-user] Efficient file reading

Stéfan van der Walt stefan@sun.ac...
Tue Jan 6 16:24:01 CST 2009


Hi Lorenzo

2009/1/7 Lorenzo Isella <lorenzo.isella@gmail.com>:
> I sometimes need to read rather large data files (~500Mb).
> These are plain text files (usually tables with 500 x 2e5 entries).
> It seems to me (but I have not done any serious test/benchmark) that R
> is faster than Python to read/write files.

For simply formatted files, numpy.fromfile should do the trick, and is
fast.  Otherwise, try numpy.loadtxt.

Cheers
Stéfan


More information about the SciPy-user mailing list