[SciPy-dev] Use of Python memory allocator -- Wait...
oliphant at ee.byu.edu
Tue Nov 15 23:19:26 CST 2005
I know some mention was made in the past of using the Python memory
allocator for scipy. Let's not for now.
I just spent way too long tracking down a "memory leak" problem for a
user that amounted to the fact that the Python memory allocator never
releases memory to the OS. Code that was working with Numeric without
growing memory would grow memory with scipy core until it bogged down
the entire machine. After closing several memory leaks in scipy core
itself, the real problem was tracked finally to the fact that the new
array scalars defined in scipy core (in fact all of the new types) all
used the Python memory allocator for their creation. Apparently this
code was using a lot of intermediate objects.
The Python memory allocator was not re-using apparently freed memory and
was instead consuming more and more system memory as time went on.
Apparently this is a problem with the Python memory allocator and is
known about. I think some patches have been discussed, but I'm not sure
what the current state of any solution is.
For scipy core, the solution is to simply not use the Python memory
allocator. I changed all the objects to use the system allocator
(malloc and free), instead. This solved the problem the user was
So, any use of the Python memory allocator should be avoided for the
time being. In addition, the array scalars should probably do
something like Python does with its own scalars.
More information about the Scipy-dev