[Numpy-discussion] Speeding up loadtxt / savetxt
Wed Apr 28 11:52:29 CDT 2010
> Do you specifically need to store text files? NumPy's binary storage
> functions (numpy.load and save) are faster.
Yes, I know. But the files I create must be readable by an application
developed in-house at our institude, and that only supports a) ASCII files
or b) some home-grown binary format, which I hate.
> Also, an efficient reader for very simply formatted text is provided
> by numpy.fromfile.
Yes, I heard about it. But the files I have to read have comments in them,
and I didn't find a way to exclude these easily.
Time needed to read a 100M file is ~13 seconds, and to write ~5 seconds.
Which is not too bad, but also still too much ...
More information about the NumPy-Discussion