[SciPy-user] partially reading a file...
Travis E. Oliphant
Wed Aug 6 12:43:50 CDT 2008
> Travis E. Oliphant a écrit :
>> fred wrote:
>>> Let's say I want to read a (binary) file which contains a nx*ny*nz array.
>>> Is it possible to read a "sub-array" from this file, ie each block of
>>> (nx/4, ny/4, nz/4) for instance, without loading the whole file ?
>> An easy way to do this which forces the operating system to do the work
>> of partial loading is to use a memory mapped file as the source of the
>> array (i.e. a memmap array).
>> Then, selecting out a block is as simple as slicing.
> Maybe I had to mention this: the aim is to cut in several files a
> "large" data file, _bigger_ than total available memory amount.
Absolutely memory mapping still applies --- it's a perfect application
for it. But, you will probably need a 64-bit system.
Memory mapping is how the OS handles "virtual memory" which uses disk
space to increase main memory. You are just using that idea directly
with a memory mapped file.
More information about the SciPy-user