[SciPy-user] SciPy, MPI and OpenMP

Anand Patil anand.prabhakar.patil@gmail....
Wed Aug 27 05:28:01 CDT 2008


Lorenzo,
Sorry for posting to such an old thread, but I'm new to multithreaded
programming and recently struggled with Python+OpenMP, and thought you might
like to hear about my experience.

Basically I'm MUCH better off now that I've stopped using OpenMP, and
instead call a serial f2py subroutine from several threads created in Python
as in the 'handythread' example on the SciPy cookbook. The reason is simply
that OpenMP is supported by newer versions of gcc than those that were used
to compile most binary distributions of Python, so if you use OpenMP you can
get compatibility problems. I had to rebuild Python from source on every
machine I used for the OpenMP stuff.

Surely that's at least partly because of my lack of skill with gcc, but I
know that at least four people have had similar problems, and the
threading-from-Python route is easier to program anyway. It's ended up being
faster for me despite the overhead of spawning Python thread objects,
because I can use Python's superior  flexibility to safely pare each
thread's work down to the bare minimum.

If you go this route, your serial f2py subroutines just need to have the
line 'cf2py threadsafe' in them. That will make them release and reacquire
the GIL as appropriate.

Anand


On Mon, Aug 18, 2008 at 11:45 PM, Robert Kern <robert.kern@gmail.com> wrote:

> On Mon, Aug 18, 2008 at 10:00, Lorenzo Isella <lorenzo.isella@gmail.com>
> wrote:
> > Dear All,
> > I have recently attended a crash course on MPI and OpenMP. The
> > examples always involved C or Fortran code.
> > Now, I have a thought: if working on a single processor, I hardly need
> > to use pure C or pure Fortran. I usually write a Fortran code for the
> > bottlenecks and compile it with f2py to create a python module I then
> > import.
> > Hence two questions:
> > (1) Can I do something similar with many processors? E.g.: write a
> > Python code, embed some compiled Fortran code which is supposed to run
> > on many processors, get the results and come back to Python.
> >
> > Python--->Fortran on many processors--->back to Python.
> > (2)Is it also possible to directly parallelize a Python code? I heard
> > about thread locking in Python.
>
> There is a global interpreter lock (GIL) when touching Python data
> structures. If you handle the threads entirely in the Fortran code and
> never call back into Python until you are finished with the threads,
> this should not be an issue for you.
>
> In bad ASCII art:
>
> Python  |Fortran /------\   |  Python
> --------|-------<-------->--|--------
>        |        \------/   |
>
> > I did some online research, there seems to be a lot of projects trying
> > to combine Python and MPI/OpenMP, but many look rather "experimental".
> > In particular, of course, I would like to hear about SciPy and
> > parallel computing.
>
> Most of the MPI wrappers these days are fairly mature. I think some of
> the OpenMP work is still pretty new, though.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-user/attachments/20080827/7e29df05/attachment-0001.html 


More information about the SciPy-user mailing list