[Numpy-discussion] Deprecating PyDataMem_RENEW ?

David Cournapeau cournape@gmail....
Mon May 5 13:25:48 CDT 2008


On Tue, May 6, 2008 at 1:59 AM, Timothy Hochberg <tim.hochberg@ieee.org> wrote:

> I don't think you would want to do this in the core of PyArray_FromIter;
> presumably realloc can sometimes reuse the existing pointer and save on
> allocating a new chunk of memory. Since there are lots of allocations in
> fromiter, this could potentially be a big performance hit.

Looking at PyArray_FromIter, there is already a log(n) behavior for
allocation, so I am not sure it would hurt so much: I would guess that
realloc often need to allocate a new block if the size is the double
of the former pointer.

That's how realloc seems to work on FreeBSD, at least:

http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/stdlib/malloc.c?rev=1.171;content-type=text%2Fplain

> (At least I think
> so, realloc has always been kinda voodoo to me).

Once malloc is implemented, I would guess realloc to be simple, no ?
For each block of memory in the malloc chain, just check whereas there
is enough size to reuse the same block and adjacent block if any,
otherwise, free + malloc + copy, non ?

 One could use
> PyMemData_NEW/PyMemData_FREE in the final allocation to make sure that the
> data is alligned, we allready do a realloc there to dump any extra space.

I guess that would be the easiest: directly use realloc in the loop,
and use PyDataMem_NEW/FREE at the end if necessary. Using realloc is
no problem if the buffer is always freed in the same function.

> Or, possibly better, one could choose which allocation strategy to use here
> depending on whether the data was alligned or not.

I mentioned step 1, because there would be a step 2 (but maybe only
numpy 2): extending the C api functions to create arrays such as
asking for an explicit alignment is possible.

David


More information about the Numpy-discussion mailing list