[SciPy-User] Speeding things up - how to use more than one computer core

Gael Varoquaux gael.varoquaux@normalesup....
Mon Apr 8 11:59:00 CDT 2013

On Mon, Apr 08, 2013 at 07:35:30AM -0400, Neal Becker wrote:
> > There are solutions, but hardly any are easy to use. I'd even say that
> > most are very challenging, and the easiest option is to rely on memapped
> > arrays, but even that is a bit technical, and will clearly introduce
> > overhead.

> Why do you think memmaped arrays would introduce overhead? 

If you are able to instanciate the arrays that matter directly with a
memmap once and for all, I agree with you. Now, if you do something like
the example posted by the OP, in which the loop is very short-lived, then
chances are that the arrays will be allocated for the loop and
deallocated after. Then the creation of the memmap induces overhead.

> The only overhead should be if you have to add some sort of
> synchronization between writers and readers (e.g., semaphores).  The
> actual data access is as fast as any other memory access.

Granted, the data access is excellent. It's the creation/deletion that I
was talking about.


More information about the SciPy-User mailing list