[SciPy-user] shared memory machines

Gael Varoquaux gael.varoquaux@normalesup....
Mon Feb 2 04:53:16 CST 2009


On Mon, Feb 02, 2009 at 12:51:51AM -0600, Robert Kern wrote:
> On Mon, Feb 2, 2009 at 00:38, Gael Varoquaux
> <gael.varoquaux@normalesup.org> wrote:
> > I think I should write empty_shmem, to complete hide the multiprocessing
> > Array, delete my useless SharedMemArray class, integrate your number of
> > processor function, and recirculate my code, if it is OK with you. In a
> > few iterations we can propose this for integration in numpy.

> Here's mine, FWIW. It goes down directly to the multiprocessing.heap
> code underlying the Array stuff. On Windows, the objects transfer via
> pickle while under UNIX, they must be inherited. Windows mmap objects
> can be pickled while UNIX mmap objects can't. Like Sturla says, we'd
> have to use named shared memory to get around this on UNIX.

Well, you know way more than I do about this. But I fear I am
miss-understanding something. Does what you are saying means that an
'empty_shmem', that would create a multiprocessing Array, and expose it
as a numpy array, is bound to fail under windows? My experiments seem to
show that this works under Linux, and this would be a very simple way of
doing shared memory. We could have a numpy.multiprocessing, with all kind
of constructors for arrays (empty, zeros, ones, *_like, and maybe
'array') that would be shared between process.

Am I out of my mind, and will this fail utterly?

Cheers,

Gaël


More information about the SciPy-user mailing list