[SciPy-user] Multithreading cookbook entry

Anne Archibald peridot.faceted@gmail....
Thu Feb 21 10:30:51 CST 2008


On 21/02/2008, Anand Patil <anand.prabhakar.patil@gmail.com> wrote:

>  First of all, that's amazing! I've been internally railing against the
>  GIL for months. But it looks like only a portion of f is being done
>  concurrently. In fact if I comment out the 'exp(y)', I don't see any
>  speedup at all.
>
>  It makes sense that you can't malloc simultaneously from different
>  threads... but if I replace 'ones' with 'empty', the time drops
>  precipitously, indicating that most of the time taken by 'ones' is
>  spent actually filling the array with ones. It seems like you should
>  be able to do that concurrently.
>
>  So my question is, what kinds of numpy functions tend to release the
>  GIL? Is there a system to it, so that one can figure out ahead of time
>  where a speedup is likely, or do you have to try and see? Do
>  third-party f2py functions with the 'threadsafe' option release the
>  GIL?

In general, the answer is that if a C extension can function outside
the GIL, it has to explicitly release it. TBH, I'm not sure what it
has to do first to make sure the interpreter is in a safe state -
maybe nothing - but it has to explicitly declare that it's not going
to modify any interpreter state.

Many numpy functions - exp is obviously an example - do this. Others
don't. It would be useful to go through the code looking at which ones
do and don't release the GIL, and put it in their docstrings; it might
be possible to make more release the GIL. It's a pretty safe bet that
the ufuncs do; I would guess that the linear algebra functions do too.
Probably not much else.

If an extension uses ctypes, whether it releases the GIL is up to
ctypes. I would guess that it doesn't, since ctypes knows nothing
about the C function, but I have never actually used ctypes.

Anne


More information about the SciPy-user mailing list