[SciPy-User] Multiprocessing and shared memory
Sun Oct 18 13:38:24 CDT 2009
Felix Schlesinger skrev:
> I have been working on an application using scipy that solves a highly
> parallel problem. To avoid the GIL in python I used to multiprocessing
> package. The main issue I ran into is shared memory.
Gaël Varoquaux and I did some work on that some months ago. It's not as
trivial as it seems, but we have a working solution.
Basically it uses named shared memory (Sys V IPC on Unix) as buffer. The
ndarray is pickled by its kernel name, the buffer is not copied. Thus
you can quickly communicate shared memory ndarrays between processes
Note that there is a pesky memory we could get rid of on Unix: It stems
from multiprocessing using os._exit instead of sys.exit to terminate
worker processes, preventing any clean-up code from executing. The bug
is in multiprocessing, not our code. Windows refcounts shared memory
segments in the kernel and is not affected. Change any occurrence of
os._exit in multiprocessing to sys.exit and it will work just fine.
Well, it's not quite up to date for all 64 bits systems. I'll fix that
More information about the SciPy-User