[Numpy-discussion] multiprocessing shared arrays and numpy
Thu Mar 11 03:04:36 CST 2010
A Sunday 07 March 2010 20:03:21 Gael Varoquaux escrigué:
> On Sun, Mar 07, 2010 at 07:00:03PM +0000, René Dudfield wrote:
> > 1. Mmap'd files are useful since you can reuse disk cache as program
> > memory. So large files don't waste ram on the disk cache.
> I second that. mmaping has worked very well for me for large datasets,
> especialy in the context of reducing memory pressure.
As far as I know, memmap files (or better, the underlying OS) *use* all
available RAM for loading data until RAM is exhausted and then start to use
SWAP, so the "memory pressure" is still there. But I may be wrong...
More information about the NumPy-Discussion