[Numpy-discussion] Deprecating PyDataMem_RENEW ?

Anne Archibald peridot.faceted@gmail....
Tue May 6 04:19:52 CDT 2008


2008/5/5 David Cournapeau <david@ar.media.kyoto-u.ac.jp>:

>  Basically, what I have in mind is, in a first step (for numpy 1.2):
>     - define functions to allocate on a given alignement
>     - make PyMemData_NEW 16 byte aligned by default (to be compatible
>  with SSE and co).
>
>  The problem was, and still is, realloc. It is not possible to implement
>  realloc with malloc/free (in a portable way), and as such, it is not
>  possible to have an aligned realloc.

How much does this matter? I mean, what if you simply left all the
reallocs as-is? The arrays that resulted from reallocs would not
typically be aligned, but we cannot in any case expect all arrays to
be aligned.

Or, actually, depending on how you manage the alignment, it may be
possible to write a portable aligned realloc. As I understand it, a
portable aligned malloc allocates extra memory, then adds something to
the pointer to make the memory arena aligned. free() must then recover
the original pointer and call free() on it - though this can
presumably be avoided if garbage collection is smart enough. realloc()
also needs to recover the pointer to call the underlying realloc().
One answer is to ensure that at least one byte of padding is always
done at the beginning, so that the number of pad bytes used on an
aligned pointer p is accessible as ((uint8*)p)[-1]. Other schemes are
available; I have a peculiar affection for the pure-python solution of
constructing a view. In any of them, it seems to me, if free() can
recover the original pointer, so can realloc()...

I don't think any numpy function can assume that its input arrays are
aligned, since users may like to pass, say, mmap()ed hunks of a file,
over which numpy has no control of the alignment. In this case, then,
how much of a problem is it if a few stray realloc()s produce
non-aligned arrays?

Anne


More information about the Numpy-discussion mailing list