[SciPy-User] Shared memory using multiprocessing.sharedctypes
Mon Mar 15 10:31:54 CDT 2010
I am interested. Can you please show us your code ?
On Monday 15 March 2010, David Baddeley wrote:
> Will have to give it a spin. Unfortunately (or fortunately - as it was
> quite instructive from a learning perspective) I got bored and already
> knocked up my own version. It delegates all the memory management to
> multiprocessing so weighs in pretty light at only 80 lines of pure python
> & works well on 64 bit systems. Just in case anyone wants yet another
> shared memory array implementation.
> ----- Original Message ----
> From: Christopher Lee-Messer <email@example.com>
> To: firstname.lastname@example.org
> Sent: Sun, 14 March, 2010 5:45:08 AM
> Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
> David Baddeley wrote::
> > I'd like shared memory numpy arrays for some code I'm trying to
> > parallelise. The solution should ideally be cross platform. I remember
> > some earlier discussion about this and, as far as I recall Sturla had a
> > working implementation which was (unfortunately) linux only (please
> > correct me if I'm wrong).
> > Looking a bit further, multiprocessing.sharedctypes seems to have all the
> > stuff needed to handle the low level bits on at least linux and windows,
> > and it seems as though it should be relatively easy to cobble something
> > together on top of this which let you use shared memory numpy arrays
> > relatively transparently. This would have the advantage that all the
> > platform specific nasty stuff is being handled by the multiprocessing
> > module, which as it's now part of the standard library, should hopefully
> > be well maintained & has a good chance of being ported to additional
> > platforms
> > Before I jump in and try writing such a wrapper - essentially a helper
> > function for creating shared numpy arrays (easy) and some pickle handlers
> > to make sure what comes out the other end looks like an array (would need
> > to think/read a bit more, but should be doable) I wanted to make sure
> > that there's not something out there already & that I'm not going to be
> > reinventing the wheel.
> [Note, I posted this via google groups originally before realizing
> that it wasn't the official list. Sorry if you get multiple versions.
> Hi David,
> I was just in the same boat. I checked out Sturla's a few weeks ago
> and tried it out on windows xp and now linux (32bit, python 2.6 with
> numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)
> It's working very well for me in soft-realtime data acquisition
> application with sizes of 1 to 800KB.
> In case it saves you time, I recently threw together some packaging
> for the code and put it on bitbucket as a numpy-sharedmem repository
> in case it might save me or someone else some time.
> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
> to work on these, but I plan to add testing and save results on
> different platforms as my app gets used in different computers in the
> lab. Nadav Horesh mentioned that there are some limitations on the
> size of the arrays for "large" arrays and Sturla mentioned that there
> may be issue on 64bit systems.
> SciPy-User mailing list
> SciPy-User mailing list
More information about the SciPy-User