[SciPy-dev] Memory mapped files in scipy core
oliphant at ee.byu.edu
Tue Nov 22 14:40:16 CST 2005
Joe Harrington wrote:
>I'm not familiar with the mmap interface, but these insertion tricks
>sound like they solve a particularly unpleasant problem of IDL that I
>hit a lot.
>It's considered nice in CS to allocate space on the fly, since it
>keeps your allocation with the code that uses it. It's particularly
>useful if you have an unknown amount of data coming in. To that end,
>I have a routine, concat, that allows you to tack on an array to any
>side of an existing array:
>x = concat(2, x, y)
>puts x next to y in the second dimension. The arrays may have any
>dimension and shape. If they're different shapes, concat fills any
>void space in with a pad value. If I'm reading in a dataset of some
>thousands of images, I can just put that in a loop and then ask the
>final array how big it is, rather than "peeking" at some ancillary
>data to find out how much space to pre-allocate in x. The
>pre-allocation line (which has to be outside the loop in IDL, or has
>to be protected by an if) is unnecessary.
This looks like a good routine to put in scipy_core if you don't mind
I can see that resizing memory mapped arrays and inserting into them
would be useful. I'm not familiar enough with the memory-mapped-file to
understand how inserting into a memory-mapped file would work.
It looks like numarray's implementation actually copies data into
RAM-based buffers on insertion and re-sizing which would seem to negate
the benefits you speak of.
More information about the Scipy-dev