[Numpy-discussion] Simple shared arrays
Erik Rigtorp
erik@rigtorp....
Thu Dec 30 20:30:21 CST 2010
Hi,
I was trying to parallelize some algorithms and needed a writable
array shared between processes. It turned out to be quite simple and
gave a nice speed up almost linear in number of cores. Of course you
need to know what you are doing to avoid segfaults and such. But I
still think something like this should be included with NumPy for
power users.
This works by inheriting anonymous mmaped memory. Not sure if this
works on windows.
import numpy as np
import multiprocessing as mp
class shared(np.ndarray):
"""Shared writable array"""
def __new__(subtype, shape, interface=None):
size = np.prod(shape)
if interface == None:
buffer = mp.RawArray('d', size)
self = np.ndarray.__new__(subtype, shape, float, buffer)
else:
class Dummy(object): pass
buffer = Dummy()
buffer.__array_interface__ = interface
a = np.asarray(buffer)
self = np.ndarray.__new__(subtype, shape=a.shape, buffer=a)
return self
def __reduce_ex__(self, protocol):
return shared, (self.shape, self.__array_interface__)
def __reduce__(self):
return __reduce_ex__(self, 0)
Also see attached file for example usage.
Erik
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shared.py
Type: text/x-python
Size: 1363 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/numpy-discussion/attachments/20101230/c6abf3a8/attachment.py
More information about the NumPy-Discussion
mailing list