[SciPy-User] Speeding things up - how to use more than one computer core
J. David Lee
Mon Apr 8 07:44:20 CDT 2013
On 04/07/2013 12:25 PM, Gael Varoquaux wrote:
> On Sun, Apr 07, 2013 at 01:11:09PM -0400, Neal Becker wrote:
>>> Regarding the difference between processes and threads:
>>> On the other hand, sharing data between threads is much cheaper than
>>> between processes.
>> I have to take issue with this statement. Sharing data could suffer no
>> overhead at all, if you use shared memory for example.
> How do you use shared memory between processes?
> There are solutions, but hardly any are easy to use. I'd even say that
> most are very challenging, and the easiest option is to rely on memapped
> arrays, but even that is a bit technical, and will clearly introduce
I've used shared memory arrays in the past, and it's actually quite
easy. They can be created using the multiprocessing module in a couple
arr = np.frombuffer(mp_arr.get_obj())
I've wondered in the past why creating a shared memory array isn't a
single line of code using numpy, as it can be so useful.
If you can, you might want to consider writing your code in a C module
and using openMP if it works for you. I've had very good luck with that,
and it's really easy to use.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the SciPy-User