[SciPy-User] IO of large ASCII table data
Tue Aug 17 12:47:15 CDT 2010
Dan Lussier wrote:
> I am looking to read in large (many million rows) ASCII space
> separated tables into numpy arrays.
If it's all space separated, all one data type, and no comment lines or
anything like that, then np.fromfile(file, sep=' ', dtype=np.float) is
about as fast as you can get.
fromfile() is very limited for reading text files, and doesn't handle
error cases well, but if it works, it's fast.
> In consulting the documentation the most likely candidate is
> numpy.genfromtext(...). Is this function pure python or does it rely
> on a C extension as was the case with Miller's TableIO?
it's python, and not all that fast.
Christopher Barker, Ph.D.
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
More information about the SciPy-User