[SciPy-dev] Use of Python memory allocator -- Wait...

[Travis Oliphant]

Developers,

I know some mention was made in the past of using the Python memory 
allocator for scipy.  Let's not for now.

I just spent way too long tracking down a "memory leak" problem for a 
user that amounted to the fact that the Python memory allocator never 
releases memory to the OS.   Code that was working with Numeric without 
growing memory would grow memory with scipy core until it bogged down 
the entire machine.  After closing several memory leaks in scipy core 
itself, the real problem was tracked finally to the fact that the new 
array scalars defined in scipy core (in fact all of the new types) all 
used the Python memory allocator for their creation.    Apparently this 
code was using a lot of intermediate objects. 

The Python memory allocator was not re-using apparently freed memory and 
was instead consuming more and more system memory as time went on.  
Apparently this is a problem with the Python memory allocator and is 
known about.  I think some patches have been discussed, but I'm not sure 
what the current state of any solution is.

For scipy core, the solution is to simply not use the Python memory 
allocator.  I changed all the objects to use the system allocator 
(malloc and free), instead.   This solved the problem the user was 
having.  

So, any use of the Python memory allocator should be avoided for the 
time being.   In addition, the array scalars should probably do 
something like Python does with its own scalars.

-Travis