[SciPy-user] scipy.integrate and threading

Anne Archibald peridot.faceted@gmail....
Fri Jun 8 20:46:53 CDT 2007


On 06/06/07, Eugen Wintersberger <eugen.wintersberger@jku.at> wrote:
> Hi there
>  I have just started with using threads in python and want to use them
> together with the integration routines in scipy. However, there seems to
> be a serious problem with threadsafty.
> I want to run two threads which perform integration simultaneously.
> However, when I do so the program exits with a Segmentation Fault. If I
> use only one thread the program runs without problems.
> Is there any simple possibility to use scipy in threads?

Much of scipy is based on publicly-available FORTRAN routines. This
has the advantage that they tend to be numerically robust and
efficient, but it has the disadvantage that they often have interfaces
that are, ah, old-fashioned. Many of them have python wrappers that
make them somewhat more comfortable to use from python, but
thread-safety (let alone parallelism) has not always been a goal. So
sometimes, yes, some scipy routines just aren't thread-safe. It's not
documented, either, as far as I can tell.

You should also be aware of python's Global Interpreter Lock, which
doesn't help with this and which means that you often don't get the
parallelism you were hoping for.

How can you avoid these problems? Well, for the specific problem of
integrating known functions, scipy includes (for example)
scipy.integrate.quadrature which is written in scipy with a perfectly
reasonable interface that should be perfectly thread-safe. (And as a
bonus, for difficult integration problems it should do some
significantly large vector operations, which release the GIL and allow
other threads to run concurrently.) It's not a very smart integrator.

A more general solution to the problem would be to wrap your favourite
non-thread-safe scipy routines in a simple routine that made sure only
one thread was in each at a time. You could even write a little
decorator.

The ideal solution, of course, is to file bugs on the scipy trac when
you find one that isn't thread-safe, so somebody can at least put a
mutex on it inside scipy, and maybe if possible make it properly
thread-safe.

Or, hmm. Someone could write a little file full of unit tests that
check various scipy routines for thread safety. That would accelerate
the above process.

Anne


More information about the SciPy-user mailing list