[Numpy-discussion] Simple shared arrays
Thu Dec 30 20:30:21 CST 2010
I was trying to parallelize some algorithms and needed a writable
array shared between processes. It turned out to be quite simple and
gave a nice speed up almost linear in number of cores. Of course you
need to know what you are doing to avoid segfaults and such. But I
still think something like this should be included with NumPy for
This works by inheriting anonymous mmaped memory. Not sure if this
works on windows.
import numpy as np
import multiprocessing as mp
"""Shared writable array"""
def __new__(subtype, shape, interface=None):
size = np.prod(shape)
if interface == None:
buffer = mp.RawArray('d', size)
self = np.ndarray.__new__(subtype, shape, float, buffer)
class Dummy(object): pass
buffer = Dummy()
buffer.__array_interface__ = interface
a = np.asarray(buffer)
self = np.ndarray.__new__(subtype, shape=a.shape, buffer=a)
def __reduce_ex__(self, protocol):
return shared, (self.shape, self.__array_interface__)
return __reduce_ex__(self, 0)
Also see attached file for example usage.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 1363 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/numpy-discussion/attachments/20101230/c6abf3a8/attachment.py
More information about the NumPy-Discussion