[Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication
Thu Jun 16 12:30:20 CDT 2011
Den 16.06.2011 18:44, skrev Christopher Barker:
> I'll bet this is a big issue, and one I'm curious about how to
> address, I have another problem where I need to multi-process, and I'd
> love to know a way to pass data to the other process and back
> *without* going through pickle. maybe memmapped files?
Remember that os.fork is copy-on-write optimized. Often this is enough
to share data.
To get data back, a possibility is to use shared memory: just memmap
from file 0 or -1 (depending on the system) and fork.
Note that the use of fork, as opposed to multiprocessing, avoids the
pickle overhead for NumPy arrays.
On Windows this sucks, because there is no fork system call. Here we are
stuck with multiprocessing and pickle, even if we use shared memory.
More information about the NumPy-Discussion