[Numpy-discussion] multiprocessing shared arrays and numpy

Gael Varoquaux gael.varoquaux@normalesup....
Thu Mar 11 03:36:42 CST 2010


On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
> As far as I know, memmap files (or better, the underlying OS) *use* all 
> available RAM for loading data until RAM is exhausted and then start to use 
> SWAP, so the "memory pressure" is still there.  But I may be wrong...

I believe that your above assertion is 'half' right. First I think that
it is not SWAP that the memapped file uses, but the original disk space,
thus you avoid running out of SWAP. Second, if you open several times the
same data without memmapping, I believe that it will be duplicated in
memory. On the other hand, when you memapping, it is not duplicated, thus
if you are running several processing jobs on the same data, you save
memory. I am very much in this case.

Gaël


More information about the NumPy-Discussion mailing list