[Numpy-discussion] reference leacks in numpy.asarray

Lisandro Dalcin dalcinl@gmail....
Thu Aug 2 17:20:22 CDT 2007


I think the problem is in  _array_fromobject (seen as numpy.array in Python)

This function parses its arguments by using the convertor
PyArray_DescrConverter2. which RETURNS A NEW REFERENCE!!! This
reference is never DECREF'ed.

BTW, A lesson I've learned of the pattern

if (!PyArg_ParseXXX(....)) return NULL

is that convertor functions should NEVER return new references to
PyObject*'s,  because if the conversion fails (because of latter wrong
argument), you leak a reference to the 'converted' object.

If this pattern is used everywhere in numpy, well, there are big
chances of leaking references in the case of bad args to C functions.

Regards,



On 8/2/07, Timothy Hochberg <tim.hochberg@ieee.org> wrote:
>
>
>
> On 8/2/07, Lisandro Dalcin <dalcinl@gmail.com> wrote:
> > using numpy-1.0.3, I believe there are a reference leak somewhere.
> > Using a debug build of Python 2.5.1 (--with-pydebug), I get the
> > following
> >
> > import sys, gc
> > import numpy
> >
> > def testleaks(func, args=(), kargs={}, repeats=5):
> >     for i in xrange(repeats):
> >         r1 = sys.gettotalrefcount()
> >         func(*args,**kargs)
> >         r2 = sys.gettotalrefcount()
> >         rd = r2-r1
> >         print 'before: %d, after: %d, diff: [%d]' % (r1, r2, rd)
> >
> > def npy_asarray_1():
> >     a = numpy.zeros(5, dtype=int)
> >     b = numpy.asarray(a, dtype=float)
> >     del a, b
> >
> > def npy_asarray_2():
> >     a = numpy.zeros(5, dtype=float)
> >     b = numpy.asarray(a, dtype=float)
> >     del a, b
> >
> > if __name__ == '__main__':
> >     testleaks(npy_asarray_1)
> >     testleaks(npy_asarray_2)
> >
> >
> > $ python npyleaktest.py
> > before: 84531, after: 84532, diff: [1]
> > before: 84534, after: 84534, diff: [0]
> > before: 84534, after: 84534, diff: [0]
> > before: 84534, after: 84534, diff: [0]
> > before: 84534, after: 84534, diff: [0]
> > before: 84531, after: 84533, diff: [2]
> > before: 84535, after: 84536, diff: [1]
> > before: 84536, after: 84537, diff: [1]
> > before: 84537, after: 84538, diff: [1]
> > before: 84538, after: 84539, diff: [1]
> >
> > It seems npy_asarray_2() is leaking a reference. I am  missing
> > something here?. The same problem is found in C, using
> > PyArray_FROM_OTF (no time to go inside to see what's going on, sorry)
> >
> > If this is know and solved in SVN, please forget me.
>
> I don't have a debug build handy to test this on, but this might not be a
> reference leak. Since you are checking the count before and after each
> cycle, it could be that there are cycles being created that are subsequently
> cleaned up by the garbage collector.
>
> Can you try instead to look at the difference between the reference count at
> the end of each cycle with the reference count before the first cycle? If
> that goes up indefinitely, then it's probably a leak. If it bounces around
> or levels off, then probably not. You'd probably want to run a bunch of
> repeats just to be sure.
> regards,
> -tim
>
> > Regards,
> >
> >
> > --
> > Lisandro Dalcín
> > ---------------
> > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> > PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> > Tel/Fax: +54-(0)342-451.1594
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion@scipy.org
> >
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
>
> --
> .  __
> .   |-\
> .
> .   tim.hochberg@ieee.org
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


More information about the Numpy-discussion mailing list