[SciPy-User] IO of large ASCII table data

Keith Goodman kwgoodman@gmail....
Tue Aug 17 13:54:28 CDT 2010


On Tue, Aug 17, 2010 at 11:43 AM, Éric Depagne <edepagne@lcogt.net> wrote:
>>
>> Here's the time it takes to read from my home directory (regular HD):
>> >> f = h5py.File('/home/kg/speed.hdf5')
>> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1
>>
>> 0.0879709720612
>>
>> And here's the time it takes to read from my ram disk:
>> >> f = h5py.File('/dev/shm/speed.hdf5')
>> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1
>>
>> 0.086874961853
>>
> That's impressive.
> Any ascii file can be read with h5py?
> And the can we "convert" the data read into numpy arrays ?

h5py can only read hdf5 files, so you'd have to convert the ascii file
to hdf5. Might be worth it if you read the same ascii file many times.
Even better would be to change the program that is writing the ascii
files to write hdf5 files.

To convert from ascii to hdf5:

arr = np.fromfile(...)
f = h5py.File('converted.hdf5')
f['arr'] = arr


More information about the SciPy-User mailing list