[SciPy-dev] advanced memory allocator?
cimrman3 at ntc.zcu.cz
Wed Nov 23 01:53:56 CST 2005
Following recent discussions here about malloc vs. python allocator and
about hunting memory related problems like buffer overruns and memory
leaks, I think a better memory allocator could help. Years ago I have
looked at the allocator of PETSc and, inspired by it, wrote a much
simpler but usable version, which can be used to replace malloc/free.
All it does is:
- for each data allocation it records the line, function, file and dir
where it occurred, together with allocated size, in a header prepended
to the data buffer
- record current/max. memory usage
- all currently dynamically allocated memory can be printed or freed
(brutal version of garbage collection :)
- memory can be checked for integrity, thanks to a 'magic cookie' value
in the header - many buffer overruns can be detected this way
- all you pay for this is a little extra space for each allocation call
Then any python extension module using the above can be at any moment
examined and its allocated memory checked, printed, freed etc.
Of course, valgrind gives you much more, but this can be used all the
time since it does not cause any performance problems...
There is one problem though: it is tied to 32-bit systems, so a 64-bit
expert would have to take a look...
I will upload it into the sandbox, if some interest arises.
More information about the Scipy-dev