[Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB

Charles R Harris charlesr.harris@gmail....
Thu Jul 23 08:16:05 CDT 2009


On Thu, Jul 23, 2009 at 5:36 AM, Kim Hansen <slaunger@gmail.com> wrote:

> OS. Win XP SP3, 32 bits
> Python: 2.5.4
> Numpy: 1.3.0
>
> I have am having some major problems converting a 750 MB recarray into
> a 850 MB recarray
>
> To save RAM I would like to use a read-only and a writeable memap for
> the two recarrays during the conversion.
>
> So I do something like:
>
> import os
> from stat import ST_SIZE
> import numpy as np
> ...
> records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize
> toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype,
> mode="r").view(np.recarray)
> result = np.memmap(result_path, dtype = result_dtype, mode = "w+",
> shape=(records,))
>
> The code manages to create the toconvert memmap (750 MB),
> but when trying to create the second memap object I get
>  File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226,
> in __new__
>    mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> WindowsError: [Error 8] Not enough storage is available to process this
> command
>
> By tracing before and after, I can see the file size is zero before
> calling mmap.mmap and has the expected 850 MB size after the
> WindowsError has been thrown somewhere inside mmap.mmap. There is 26
> GB of free disc space, so the error message seems wrong?
>
> If I comment out the creation of the first memmap, I can successfully
> create the result memmap, so the error seems to be related to the
> accumulated size of all mmap.mmaps generated. I have other cases with
> somewhat smaller files to convert where the transition is OK. It seem
> like I begin to get these problems when the accumulated size of
> memmaps exceeds 1GB
>
> I am surprised by this, as
>
>
> http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#memory-mapped-file-arrays
>
> mentions there are upper bounds to the size when using versions of
> python before 2.5.
>
> >From this I had the impression that there was no size limit as long as
> you were using ver. >=2.5 (as I am)
>
> Is it due to the 32 bit OS I am using?
>

It could be. IIRC, 32 bit windows gives user programs 2 GB of addressable
memory, so your files need to fit in that space even if the data is on disk.
You aren't using that much memory but you are close and it could be that
other programs make up the difference. Maybe you can monitor the memory to
get a better idea of the usage.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090723/522b1351/attachment.html 


More information about the NumPy-Discussion mailing list