[Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject
Dan S
dan.s.towell+numpy@gmail....
Mon Apr 20 05:44:14 CDT 2009
Thanks all for responses. Continuation below:
2009/4/18 Charles R Harris <charlesr.harris@gmail.com>:
>
>
> On Fri, Apr 17, 2009 at 9:25 AM, Dan S <dan.s.towell+numpy@gmail.com> wrote:
>>
>> Hi -
>>
>> I have written a numpy extension which works fine but has a memory
>> leak. It takes a single array argument and returns a single scalar.
>> After reducing the code down in order to chase the problem, I have the
>> following:
>>
>> static PyObject * kdpee_pycall(PyObject *self, PyObject *args)
>> {
>> PyObject *input;
>> PyArrayObject *array;
>> int n, numdims;
>>
>> if (!PyArg_ParseTuple(args, "O", &input))
>> return NULL;
>>
>> // Ensure we have contiguous, 2D, floating-point data:
>> array = (PyArrayObject*) PyArray_ContiguousFromObject(input,
>> PyArray_DOUBLE, 2, 2);
>>
>> if(array==NULL){
>> printf("kdpee_py: nullness!\n");
>> return NULL;
>> }
>> PyArray_XDECREF(array);
>> Py_DECREF(array); // destroy the contig array
>> Py_DECREF(input); // is this needed? doc says no, but it seems to fix
> ^^^^^^^^^^^^^^^^^^^^
>
> Shouldn't be, but there might be a bug somewhere which causes the reference
> count of input to be double incremented. Does the reference count in the
> test script increase without this line?
Yes - if I comment out Py_DECREF(input), then sys.getrefcount(a) goes
from 2, to 5002, to 10002, every time I run that little 5000-fold test
iteration.
(Py_DECREF(input) is not suggested in the docs - I tried it in
desperation and for some reason it seems to reduce the problem, since
it gets rid of the refleak, although it doesn't solve the memleak.)
>> return PyFloat_FromDouble(3.1415927); // temporary
>> }
>>
>>
>> The test code is
>>
>> from numpy import *
>> from kdpee import *
>> import sys
>> a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]])
>> sys.getrefcount(a)
>> for i in range(5000): tmp = kdpee(a)
>>
>> sys.getrefcount(a)
>>
>> Every time I run this code I get "2" back from both calls to
>> sys.getrefcount(a), but I'm still getting a memory leak of about 0.4
>> MB. What am I doing wrong?
>
> So about 80 bytes/iteration? How accurate is that .4 MB? does it change with
> the size of the input array?
OK, pinning it down more precisely I get around 86--90 bytes per
iteration (using ps, which gives me Kb resolution). It does not change
if I double or quadruple the size of the input array.
Today I tried using "heapy": using the heap() and heapu() methods, I
find no evidence of anything suspicious - there is no garbage left
lying on the heap of python objects. So it's presumably something
deeper. But as you can see, my C code doesn't perform any malloc() or
suchlike, so I'm stumped.
I'd be grateful for any further thoughts.
Thanks
Dan
More information about the Numpy-discussion
mailing list