[SciPy-User] IO of large ASCII table data
Tue Aug 17 13:51:23 CDT 2010
On Aug 17, 2010, at 2:41 PM, Dan Lussier wrote:
> That's great. Thanks.
> I am going to give np.fromfile(...) a try as I have data that is
> pretty uniform and will fixup the output as necessary.
> On my system reading a 1.2M row by 11 column data file
> np.genfromtxt(...) took 50 seconds while np.fromfile(...) plus a
> np.reshape(...) to change the shape of the output took under 2
There are multiple loops involved in genfromtxt that you don't have in fromtxt. That's unfortunately the price to pay for flexibility (ie, you don't know what your array should look like beforehand, and don't know want to deal w/ missing values)...
More information about the SciPy-User