[Numpy-discussion] How to resize numpy.memmap?

[Pearu Peterson]

Hi again,
To answer to the second part of my question, here follows an example
demonstrating how to "resize" a memmap:

>>> fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
>>> fp._mmap.resize(11)
>>> cp = numpy.ndarray.__new__(numpy.memmap, (fp._mmap.size(),), dtype=fp.dtype, buffer=fp._mmap, offset=0, order='C')
>>> cp[-1] = 99
>>> cp[1] = 33
>>> cp
    memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0, 99], dtype=uint8)
>>> fp
    memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0], dtype=uint8)
>>> del fp, cp
>>> fp = numpy.memmap('test.dat', mode='r')
>>> fp
    memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0, 99], dtype=uint8)

Would there be any interest in turning the above code to numpy.memmap
method, say, to resized(newshape)? For example, for resolving the original
problem, one could have

fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
fp = fp.resized(11)

Regards,
Pearu

On Sun, Jun 6, 2010 at 10:19 PM, Pearu Peterson
<pearu.peterson@gmail.com> wrote:
> Hi,
>
> I am creating a rather large file (typically 100MBi-1GBi) with numpy.memmap
> but in some cases the initial estimate to the file size is just few
> bytes too small.
> So, I was trying to resize the memmap with a failure as demonstrated
> with the following
> example:
>
>>>> fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
>>>> fp.resize(11, refcheck=False)
> ...
> ValueError: cannot resize this array:  it does not own its data
>
> My question is, is there a way to "fix" this or may be there exist some other
> technique to resize memmap. I have tried resizing memmap's _mmap attribute
> directly:
>
>>>> fp._mmap.resize(11)
>>>> fp._mmap.size()
>    11
>
> but the size of memmap instance remains unchanged:
>>>> fp.size
>    10
>
> Thanks,
> Pearu
>