[SciPy-user] shared memory machines

Sturla Molden sturla@molden...
Thu Feb 12 08:53:12 CST 2009


On 2/11/2009 3:24 PM, Gael Varoquaux wrote:

> I have put in some more print statements. I have the fealing that it is
> not cleaned up. I am attaching my test code, and the modified
> sharedmemory_sysv.pyx for debug. The output is the following:

I have reproduced the same error in Windows. A suspected, it stems from 
using multiprocessing's Finalizer. For some reason it is never called. 
Also having the heap object as a class attribute of the BufferWrapper 
prevents clean-up with a __del__ method. Whereas having it as a global 
variable in the module works ok. The problem is not the Cython extension 
code, it is the array_heap.py module.

I think we should not use a malloc at all. If a segment cannot be reused 
(Heap.free) until all other handles to it is closed. This is a bug in my 
code. The easiest solution is to remove the heap malloc all together. In 
that case, the allocator for small arrays will just reuse a shared 
segment until it is exhausted, and then discared it. Large arrays will 
get their own segment. Otherwise we have to do refcounting for open 
handles (manually in Windows) and we are back to thread-based cleanup in 
the creator.

Sturla Molden




More information about the SciPy-user mailing list