[SciPy-User] Shared memory using multiprocessing.sharedctypes

Christopher Lee-Messer chris.leemesser@gmail....
Tue Mar 16 01:27:19 CDT 2010


Excellent, and many thanks.  I've learned a lot by looking at your
code.  I didn't realize how much good stuff is in multiprocessing.
-chris

On Mon, Mar 15, 2010 at 3:40 PM, David Baddeley
<david_baddeley@yahoo.com.au> wrote:
> Here's the code, hope it's useful.
>
> cheers,
> David
>
> --- On Tue, 16/3/10, Charles Vejnar <ce@vejnar.eu> wrote:
>
>> From: Charles Vejnar <ce@vejnar.eu>
>> Subject: Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes
>> To: "SciPy Users List" <scipy-user@scipy.org>
>> Cc: "David Baddeley" <david_baddeley@yahoo.com.au>
>> Received: Tuesday, 16 March, 2010, 4:31 AM
>> Hi David,
>>
>> I am interested. Can you please show us your code ?
>>
>> BW,
>> Charles
>>
>> On Monday 15 March 2010, David Baddeley wrote:
>> > Thanks!
>> >
>> > Will have to give it a spin. Unfortunately  (or
>> fortunately - as it was
>> >  quite instructive from a learning perspective) I
>> got bored and already
>> >  knocked up my own version. It delegates all the
>> memory management to
>> >  multiprocessing so weighs in pretty light at
>> only 80 lines of pure python
>> >  & works well on 64 bit systems. Just in case
>> anyone wants yet another
>> >  shared memory array implementation.
>> >
>> > cheers,
>> > David
>> >
>> >
>> > ----- Original Message ----
>> > From: Christopher Lee-Messer <chris.leemesser@gmail.com>
>> > To: scipy-user@scipy.org
>> > Sent: Sun, 14 March, 2010 5:45:08 AM
>> > Subject: [SciPy-User]  Shared memory using
>> multiprocessing.sharedctypes
>> >
>> > David Baddeley wrote::
>> > > I'd like shared memory numpy arrays for some code
>> I'm trying to
>> > > parallelise. The solution should ideally be cross
>> platform. I remember
>> > > some earlier discussion about this and, as far as
>> I recall Sturla had a
>> > > working implementation which was (unfortunately)
>> linux only (please
>> > > correct me if I'm wrong).
>> > >
>> > > Looking a bit further,
>> multiprocessing.sharedctypes seems to have all the
>> > > stuff needed to handle the low level bits on at
>> least linux and windows,
>> > > and it seems as though it should be relatively
>> easy to cobble something
>> > > together on top of this which let you use shared
>> memory numpy arrays
>> > > relatively transparently. This would have the
>> advantage that all the
>> > > platform specific nasty stuff is being handled by
>> the multiprocessing
>> > > module, which as it's now part of the standard
>> library, should hopefully
>> > > be well maintained & has a good chance of
>> being ported to additional
>> > > platforms
>> > >
>> > > Before I jump in and try writing such a wrapper -
>> essentially a helper
>> > > function for creating shared numpy arrays (easy)
>> and some pickle handlers
>> > > to make sure what comes out the other end looks
>> like an array (would need
>> > > to think/read a bit more, but should be doable) I
>> wanted to make sure
>> > > that there's not something out there already
>> & that I'm not going to be
>> > > reinventing the wheel.
>> >
>> > [Note, I posted this via google groups originally
>> before realizing
>> > that it wasn't the official list. Sorry if you get
>> multiple versions.
>> > -chris]
>> >
>> > Hi David,
>> > I was just in the same boat.  I checked out
>> Sturla's a few weeks ago
>> > and tried it out on windows  xp and now linux
>> (32bit, python 2.6 with
>> > numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with
>> numpy 1.2.1)
>> >
>> > It's working very well for me in soft-realtime data
>> acquisition
>> > application with sizes of 1 to 800KB.
>> >
>> > In case it saves you time, I recently threw together
>> some packaging
>> > for the code and put it on bitbucket as a
>> numpy-sharedmem repository
>> > in case it might save me or someone else some time.
>> >
>> > http://bitbucket.org/cleemesser/numpy-sharedmem/
>> >
>> > I don't know if Sturla, Gael Varoquaux, or Robert Kern
>> are continuing
>> > to work on these, but I plan to add testing and save
>> results on
>> > different platforms as my app gets used in different
>> computers in the
>> > lab.  Nadav Horesh mentioned that there are some
>> limitations on the
>> > size of the arrays for "large" arrays and Sturla
>> mentioned that there
>> > may be issue on 64bit systems.
>> > -Chris
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>



-- 
Christopher Lee-Messer MD PhD
Instructor in Pediatric Neurology
Postdoctoral Fellow - Deisseroth Lab
Stanford Medical Center
chris (at) lee-messer.net


More information about the SciPy-User mailing list