[Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject
Mon Apr 20 05:44:14 CDT 2009
Thanks all for responses. Continuation below:
2009/4/18 Charles R Harris <firstname.lastname@example.org>:
> On Fri, Apr 17, 2009 at 9:25 AM, Dan S <email@example.com> wrote:
>> Hi -
>> I have written a numpy extension which works fine but has a memory
>> leak. It takes a single array argument and returns a single scalar.
>> After reducing the code down in order to chase the problem, I have the
>> static PyObject * kdpee_pycall(PyObject *self, PyObject *args)
>> PyObject *input;
>> PyArrayObject *array;
>> int n, numdims;
>> if (!PyArg_ParseTuple(args, "O", &input))
>> return NULL;
>> // Ensure we have contiguous, 2D, floating-point data:
>> array = (PyArrayObject*) PyArray_ContiguousFromObject(input,
>> PyArray_DOUBLE, 2, 2);
>> printf("kdpee_py: nullness!\n");
>> return NULL;
>> Py_DECREF(array); // destroy the contig array
>> Py_DECREF(input); // is this needed? doc says no, but it seems to fix
> Shouldn't be, but there might be a bug somewhere which causes the reference
> count of input to be double incremented. Does the reference count in the
> test script increase without this line?
Yes - if I comment out Py_DECREF(input), then sys.getrefcount(a) goes
from 2, to 5002, to 10002, every time I run that little 5000-fold test
(Py_DECREF(input) is not suggested in the docs - I tried it in
desperation and for some reason it seems to reduce the problem, since
it gets rid of the refleak, although it doesn't solve the memleak.)
>> return PyFloat_FromDouble(3.1415927); // temporary
>> The test code is
>> from numpy import *
>> from kdpee import *
>> import sys
>> a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]])
>> for i in range(5000): tmp = kdpee(a)
>> Every time I run this code I get "2" back from both calls to
>> sys.getrefcount(a), but I'm still getting a memory leak of about 0.4
>> MB. What am I doing wrong?
> So about 80 bytes/iteration? How accurate is that .4 MB? does it change with
> the size of the input array?
OK, pinning it down more precisely I get around 86--90 bytes per
iteration (using ps, which gives me Kb resolution). It does not change
if I double or quadruple the size of the input array.
Today I tried using "heapy": using the heap() and heapu() methods, I
find no evidence of anything suspicious - there is no garbage left
lying on the heap of python objects. So it's presumably something
deeper. But as you can see, my C code doesn't perform any malloc() or
suchlike, so I'm stumped.
I'd be grateful for any further thoughts.
More information about the Numpy-discussion