[Numpy-discussion] Simple shared arrays

[Erik Rigtorp]

Hi,

I was trying to parallelize some algorithms and needed a writable
array shared between processes. It turned out to be quite simple and
gave a nice speed up almost linear in number of cores. Of course you
need to know what you are doing to avoid segfaults and such. But I
still think something like this should be included with NumPy for
power users.

This works by inheriting anonymous mmaped memory. Not sure if this
works on windows.

import numpy as np
import multiprocessing as mp

class shared(np.ndarray):
    """Shared writable array"""

    def __new__(subtype, shape, interface=None):
        size = np.prod(shape)
        if interface == None:
            buffer = mp.RawArray('d', size)
            self = np.ndarray.__new__(subtype, shape, float, buffer)
        else:
            class Dummy(object): pass
            buffer = Dummy()
            buffer.__array_interface__ = interface
            a = np.asarray(buffer)
            self = np.ndarray.__new__(subtype, shape=a.shape, buffer=a)
        return self

    def __reduce_ex__(self, protocol):
        return shared, (self.shape, self.__array_interface__)

    def __reduce__(self):
        return __reduce_ex__(self, 0)

Also see attached file for example usage.

Erik
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shared.py
Type: text/x-python
Size: 1363 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/numpy-discussion/attachments/20101230/c6abf3a8/attachment.py