[SciPy-user] Multithreading cookbook entry

Lou Pecora lou_boog2000@yahoo....
Thu Feb 21 11:51:43 CST 2008


--- Anne Archibald <peridot.faceted@gmail.com> wrote:

> In general, the answer is that if a C extension can
> function outside
> the GIL, it has to explicitly release it. TBH, I'm
> not sure what it
> has to do first to make sure the interpreter is in a
> safe state -
> maybe nothing - but it has to explicitly declare
> that it's not going
> to modify any interpreter state.
> 
> Many numpy functions - exp is obviously an example -
> do this. Others
> don't. It would be useful to go through the code
> looking at which ones
> do and don't release the GIL, and put it in their
> docstrings; it might
> be possible to make more release the GIL. It's a
> pretty safe bet that
> the ufuncs do; I would guess that the linear algebra
> functions do too.
> Probably not much else.
> 
> If an extension uses ctypes, whether it releases the
> GIL is up to
> ctypes. I would guess that it doesn't, since ctypes
> knows nothing
> about the C function, but I have never actually used
> ctypes.

Anne,  Thanks for your answers.  They are helping, but
I'm still vague on the GIL.  I have a few more
questions, two on your handythread.py code and one on
releasing the GIL for a C extension.  Thanks for you
patience and help. BTW, I have a MacBook Pro with 2
CPUs.

(1) In your code if return_ = True I get a return
value from the foreach function only when nthreads>1,
but not when nthreads=1.  Looking at the code the
nthreads=1 ends up in the else: at the bottom which
looks like:

	else:
		if return_:
			for v in l:
				f(v)
		else:
			return

and is puzzling.  Nothing is returned in the if part
and f is not even called in the else part.  Is this a
bug?

(2) If I replace the sleep(0.5) call in your f
function with a loop that just does a simple
calculation to eat up time, then in the call to
foreach when nthreads=2 the time to run the code goes
up by factors of ~100 or so.  I'm guessing here that
it's because the GIL is not release for my version,
but is release in the sleep(0.5) function in your
version.  Is that right?

(3) You mention that ctypes probably doesn't release
the GIL. I would guess that too, since it would be
dangerous as I (vaguely) understand the GIL.  But does
the GIL have to be released in the Cextension or can
it be release in the step just before I call the C
extension from Python? I.e. is release on the Python
side possible?  If not, I guess I will have to look
over the numpy code as you suggest.  If possible, I
suppose the GIL must be enabled immediately on return
from the C extension.

Thanks, again.



-- Lou Pecora,   my views are my own.


      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping


More information about the SciPy-user mailing list