[SciPy-User] Multiprocessing and shared memory
Sun Oct 18 19:04:55 CDT 2009
Sturla Molden wrote:
> Felix Schlesinger skrev:
>> I have been working on an application using scipy that solves a highly
>> parallel problem. To avoid the GIL in python I used to multiprocessing
>> package. The main issue I ran into is shared memory.
> Gaël Varoquaux and I did some work on that some months ago. It's not as
> trivial as it seems, but we have a working solution.
> Basically it uses named shared memory (Sys V IPC on Unix) as buffer. The
> ndarray is pickled by its kernel name, the buffer is not copied. Thus
> you can quickly communicate shared memory ndarrays between processes
> (using multiprocessing.Queue).
Wouldn't posix ipc be nicer than sysv?
More information about the SciPy-User