[SciPy-dev] Memory mapped files in scipy core

[Travis Oliphant]

Joe Harrington wrote:

>I have hit the 2GB limit recently, but fortunately only have 2 32-bit
>machines left.  The 2GB limit does put a damper on things!  You'll see
>a lot more call for mmapped files in the coming years.  For the
>reasons given above, I think that will be true even if RAM does grow
>with dataset size.
>  
>

The 2GB limit is of course an issue with Python's memory-mapped code.  
The Python memory map code definitely needs some attention in order to 
be more useful.  

Just to clarify, there is now a memmap call in scipy_core that creates a 
(sub-class) of the ndarray using a memory-mapped file for the buffer.   
You can manipulate it just like an array object and I could see adding a 
simple resize function as well (provided nobody else was using it's buffer).

-Travis