[SciPy-user] Multithreading cookbook entry
Thu Feb 21 10:30:51 CST 2008
On 21/02/2008, Anand Patil <firstname.lastname@example.org> wrote:
> First of all, that's amazing! I've been internally railing against the
> GIL for months. But it looks like only a portion of f is being done
> concurrently. In fact if I comment out the 'exp(y)', I don't see any
> speedup at all.
> It makes sense that you can't malloc simultaneously from different
> threads... but if I replace 'ones' with 'empty', the time drops
> precipitously, indicating that most of the time taken by 'ones' is
> spent actually filling the array with ones. It seems like you should
> be able to do that concurrently.
> So my question is, what kinds of numpy functions tend to release the
> GIL? Is there a system to it, so that one can figure out ahead of time
> where a speedup is likely, or do you have to try and see? Do
> third-party f2py functions with the 'threadsafe' option release the
In general, the answer is that if a C extension can function outside
the GIL, it has to explicitly release it. TBH, I'm not sure what it
has to do first to make sure the interpreter is in a safe state -
maybe nothing - but it has to explicitly declare that it's not going
to modify any interpreter state.
Many numpy functions - exp is obviously an example - do this. Others
don't. It would be useful to go through the code looking at which ones
do and don't release the GIL, and put it in their docstrings; it might
be possible to make more release the GIL. It's a pretty safe bet that
the ufuncs do; I would guess that the linear algebra functions do too.
Probably not much else.
If an extension uses ctypes, whether it releases the GIL is up to
ctypes. I would guess that it doesn't, since ctypes knows nothing
about the C function, but I have never actually used ctypes.
More information about the SciPy-user