[SciPy-user] Multithreading cookbook entry
Thu Feb 21 14:09:03 CST 2008
Sorry for the huge picture, really ignorant question (my specialty !) but why couldn't all this be encapsulated in something like a version of MPI for python (since I don't know anything about the GIL I assume that might be the answer) - I recall years ago that Sun had a beautiful design (amazing !) re. their version of MPI that ran on their NUMA architecture - the library allowed the code to know whether one was communicating/running in shared and/or distributed memory and dispatched things appropriately (i.e. threaded or not). I know this is a bit futuristic but all of this should just be transparent to the non expert user; one should only have to learn details when parallelization is non-trivial. Maybe MPI isn't the best model for consolidation but it seems to abstract parallel operations in a reasonable way.
Center for Imaging of Neurodegenerative Disease, UCSF
VA Medical Center, MRS Unit (114M)
Phone: (415) 221-4810 x3114
FAX: (415) 668-2864
Email: karl young at ucsf edu
From: email@example.com on behalf of Anand Patil
Sent: Thu 2/21/2008 11:39 AM
To: SciPy Users List
Subject: Re: [SciPy-user] Multithreading cookbook entry
> Yep. Oops. Fixed in the v2 versions of the files. The wiki doesn't
> make a very good version control system. Is it worth incorporating
> those files into scipy?
I vote yes. In my opinion the following would combine to form a killer feature:
- The handythread idea is developed a little, maybe to provide
functionality comparable to OpenMP
- Instructions for releasing the GIL in different extension types
(swig, f2py, pyrex) are combined in one place
- The numpy functions that release the GIL are clearly enumerated.
Seriously, this is too big of a deal to be just a cookbook entry. I
spent a full week last month beating my head against OpenMP trying to
do something embarrassingly parallel in an f2py extension. I had to
apply a patch to gcc 4.2's libgomp, compile it manually, learn how
linking works, and try several other options because OpenMP was so
frustrating. Now it works but I have tons of bug-prone code
duplication in Fortran because I couldn't figure out how to just apply
the same parallelism structure to all subroutines. The ability to
multithread from Python would have saved me all of that work.
SciPy-user mailing list
More information about the SciPy-user