[SciPy-dev] Memory mapped files in scipy core
oliphant at ee.byu.edu
Tue Nov 22 14:52:38 CST 2005
Joe Harrington wrote:
>I have hit the 2GB limit recently, but fortunately only have 2 32-bit
>machines left. The 2GB limit does put a damper on things! You'll see
>a lot more call for mmapped files in the coming years. For the
>reasons given above, I think that will be true even if RAM does grow
>with dataset size.
The 2GB limit is of course an issue with Python's memory-mapped code.
The Python memory map code definitely needs some attention in order to
be more useful.
Just to clarify, there is now a memmap call in scipy_core that creates a
(sub-class) of the ndarray using a memory-mapped file for the buffer.
You can manipulate it just like an array object and I could see adding a
simple resize function as well (provided nobody else was using it's buffer).
More information about the Scipy-dev