[Numpy-discussion] Is there anyway to read raw binary file via pytable?

脑关(BrainGateway)生命科学仪器 braingateway@gmail....
Thu Jul 29 14:41:56 CDT 2010


2010/7/28 Ken Watford <kwatford+scipy@gmail.com>:
> 2010/7/28 脑关生命科学仪器 <braingateway@gmail.com>:
>> it seems like pytable only support HDF5. I had some 500GB numerical arrays
>> to process. Pytable claims to have some advance feature to enhance
>> processing speed and largely reduce physical memory requirement. However, I
>> do not wanna touch the raw data I had. Simply because I do not have doubled
>> diskspace to covert all 10TB data into HDF5. Is there any way to let pytable
>> read raw binary files or alternatively to package raw files into HDF5 format
>> without change the files themselves.?
>>
>> Thanks
>>
>> Brain Gateway
>
> HDF5 does support datasets with an external contiguous binary file as
> the storage area. For documentation on this, see:
> http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html#Allocation

Thanks a lot!
I do found 3 API functions to handle this problem:
H5P have 3 function to handle external raw data file.
H5Pset_external
http://davis.lbl.gov/Manuals/HDF5-1.6.1/RM_H5P.html#Property-SetExternal
H5Pget_external_count
http://davis.lbl.gov/Manuals/HDF5-1.6.1/RM_H5P.html#Property-GetExternalCount
H5Pget_external
http://davis.lbl.gov/Manuals/HDF5-1.6.1/RM_H5P.html#Property-GetExternal

However, right now h5py does not provide these functions, though
existing datasets created with
external storage should work fine.

I need to use C, fortran, or matlab to generate datasets with external
data. So far it works fine.

LittleBigBrain


More information about the NumPy-Discussion mailing list