[Numpy-discussion] Interesting memory leak

David Grant davidgrant at gmail.com
Thu Aug 17 18:25:12 CDT 2006


Hello all,

I had a massive memory leak in some of my code. It would basically end up
using up all 1GB of my RAM or more if I don't kill the application. I
managed to finally figure out which portion of the code was causing the leak
(with great difficulty) and have a little example which exposes the leak. I
am using numpy-0.9.8 and I'm wondering if perhaps this is already fixed in
1.0b2. Run this through valgrind with appropriate options (I used the
recommended valgrind_py.sh that I found on scipy's site somewhere) and this
will leak 100kB. Increase the xrange on the big loop and you can watch the
memory increase over time in top.

The interesting thing is that the only difference between the leaky and
non-leaky code is:
if not adjacencyMatrix[anInt2,anInt1] == 0: (leaky)
vs.
if not adjacencyMatrix[anInt2][anInt1] == 0: (non-leaky)

however another way to make the leaky code non-leaky is to change
anArrayOfInts to just be [1]

Here's the code:


from numpy import array

def leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
    ys = set()
    for aList in aListOfArrays:
        for anInt1 in anArrayOfInts:
            for anInt2 in aList:
                if not adjacencyMatrix[anInt2,anInt1] == 0:
                    ys.add(anInt1)
    return ys

def nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
    ys = set()
    for aList in aListOfArrays:
        for anInt1 in anArrayOfInts:
            for anInt2 in aList:
                if not adjacencyMatrix[anInt2][anInt1] == 0:
                    ys.add(anInt1)
    return ys

if __name__ == "__main__":
    for i in xrange(10000):
        aListOfArrays = [[0, 1]]
        anArrayOfInts = array([1])
        adjacencyMatrix = array([[0,1],[1,0]])
#COMMENT OUT ONE OF THE 2 LINES BELOW
        #bar = nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)
        bar = leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)



-- 
David Grant
http://www.davidgrant.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20060817/db9d3416/attachment.html 


More information about the Numpy-discussion mailing list