[Numpy-discussion] Is there anyway to read raw binary file via pytable?

Ken Watford kwatford+scipy@gmail....
Wed Jul 28 15:37:57 CDT 2010


2010/7/28 脑关生命科学仪器 <braingateway@gmail.com>:
> it seems like pytable only support HDF5. I had some 500GB numerical arrays
> to process. Pytable claims to have some advance feature to enhance
> processing speed and largely reduce physical memory requirement. However, I
> do not wanna touch the raw data I had. Simply because I do not have doubled
> diskspace to covert all 10TB data into HDF5. Is there any way to let pytable
> read raw binary files or alternatively to package raw files into HDF5 format
> without change the files themselves.?
>
> Thanks
>
> Brain Gateway

HDF5 does support datasets with an external contiguous binary file as
the storage area. For documentation on this, see:
http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html#Allocation

Whether or not you can easily use this fact from Python is another
issue. You might check h5py's documentation to see if it can modify
the necessary properties.


More information about the NumPy-Discussion mailing list