[Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB
Thu Jul 23 06:36:24 CDT 2009
OS. Win XP SP3, 32 bits
I have am having some major problems converting a 750 MB recarray into
a 850 MB recarray
To save RAM I would like to use a read-only and a writeable memap for
the two recarrays during the conversion.
So I do something like:
from stat import ST_SIZE
import numpy as np
records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize
toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype,
result = np.memmap(result_path, dtype = result_dtype, mode = "w+",
The code manages to create the toconvert memmap (750 MB),
but when trying to create the second memap object I get
File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226,
mm = mmap.mmap(fid.fileno(), bytes, access=acc)
WindowsError: [Error 8] Not enough storage is available to process this command
By tracing before and after, I can see the file size is zero before
calling mmap.mmap and has the expected 850 MB size after the
WindowsError has been thrown somewhere inside mmap.mmap. There is 26
GB of free disc space, so the error message seems wrong?
If I comment out the creation of the first memmap, I can successfully
create the result memmap, so the error seems to be related to the
accumulated size of all mmap.mmaps generated. I have other cases with
somewhat smaller files to convert where the transition is OK. It seem
like I begin to get these problems when the accumulated size of
memmaps exceeds 1GB
I am surprised by this, as
mentions there are upper bounds to the size when using versions of
python before 2.5.
>From this I had the impression that there was no size limit as long as
you were using ver. >=2.5 (as I am)
Is it due to the 32 bit OS I am using?
Is there anything I can do to resolve the problem?
More information about the NumPy-Discussion