[Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related.

Jeffrey Spencer jeffspencerd@gmail....
Mon May 23 03:54:09 CDT 2011


I realized if I re-build numpy it takes the interpreter compile options or you can edit site.cfg.

I actually solved the issue by getting the newest svn copy of numpy. It must be something with 1.6b2. The newest SVN has no issues with either library and calls to sum or prod. I was getting memory leak with both the tcmalloc and malloc with the 1.6b2 version of numpy.




There are a few places where we (improperly) directly call malloc()
instead of PyMem_Malloc(), so yes, you should rebuild numpy against
TCMalloc in addition to the Python interpreter.

How do I build against the -ltcmalloc library? Just not sure how to 
input compile time arguments with the build script.

from numpy import arange, sum
>
>  for x in range(10000000):
>            inhibVal = sum(arange(15))

Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy
1.5.1 package from ubuntu, and using 1.6.1.dev-a265004.


efiring@manini:~$ uname -a
Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux


Eric

Also noticed that the same issue occurs as described with using the prod 
instead of sum. Other commands like copy and reshape have no issues. I 
have a machine with 11.04 on it so I will give that a go later to see 
what goes.

Thanks, Jeff

Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern 
<robert.kern@gmail.com> Subject: Re: [Numpy-discussion] Python memory 
management issues using Linux. Maybe Numpy related. To: Discussion of 
Numerical Python <numpy-discussion@scipy.org> Message-ID: 
<BANLkTikUgYm8COoZEhNP2tJGYRb_PBH1og@mail.gmail.com> Content-Type: 
text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer 
<jeffspencerd@gmail.com> wrote:

> >  I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory
> >  back to the OS. I have code here the exhibits the issue:
> >
> >  http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g
> >
> >  Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/
> >  that this was a problem fixed by compiling your own version of Python
> >  using the TCMalloc library which I have now done. The issue is still
> >  exhibited so I was keen to know if possibly it was because numpy is
> >  compiled and linked to the default memory management library (not sure
> >  if this is true or how this works but would I also need to compile numpy
> >  linked to the TCMalloc library). Let me know if anyone has any insight
> >  on this issue. Or if it isn't related to numpy at all.
There are a few places where we (improperly) directly call malloc()
instead of PyMem_Malloc(), so yes, you should rebuild numpy against
TCMalloc in addition to the Python interpreter.

-- Robert Kern "I have come to believe that the whole world is an 
enigma, a harmless enigma that is made terrible by our own mad attempt 
to interpret it as though it had an underlying truth." ? -- Umberto Eco

-- 
________________________
Jeffrey Spencer
jeffspencerd@gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110523/7041c2d9/attachment.html 


More information about the NumPy-Discussion mailing list