[SciPy-User] Multiprocessing and shared memory

Neal Becker ndbecker2@gmail....
Sun Oct 18 19:04:55 CDT 2009


Sturla Molden wrote:

> Felix Schlesinger skrev:
>> I have been working on an application using scipy that solves a highly
>> parallel problem. To avoid the GIL in python I used to multiprocessing
>> package. The main issue I ran into is shared memory.
> 
> Gaël Varoquaux and I did some work on that some months ago. It's not as
> trivial as it seems, but we have a working solution.
> 
> http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip
> 
> Basically it uses named shared memory (Sys V IPC on Unix) as buffer. The
> ndarray is pickled by its kernel name, the buffer is not copied. Thus
> you can quickly communicate shared memory ndarrays between processes
> (using multiprocessing.Queue).
> 
Wouldn't posix ipc be nicer than sysv?



More information about the SciPy-User mailing list