[Numpy-discussion] Use OpenBLAS for the binary releases?

David Cournapeau cournape@gmail....
Wed Nov 21 08:37:25 CST 2012


On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden <sturla@molden.no> wrote:
> On 21.11.2012 15:01, David Cournapeau wrote:
>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla@molden.no>  wrote:
>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>
>> Because of this: https://gist.github.com/2027412
>>
>> Numpy + accelerate is essentially unusable with multiprocessing.
>
> See this too:
>
> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>
> If they put a memory leak in DGEMM, basically most of LAPACK is useless
> as well. That is very sloppy of Apple.
>
> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>
> It doesn't seem the Grand Central Dispatch is very robust then...

I don't think it has anything to do with robustness as much as
multiprocessing doing things not supported by GCD. The answer we got
from Apple when we looked into this with Olivier Grisel at pycon was:

"""
For API outside of POSIX, including GCD and technologies like
Accelerate, we do not support usage on both sides of a fork(). For
this reason among others, use of fork() without exec is discouraged in
general in processes that use layers above POSIX.

We recommend that you either restrict usage of blas to the parent or
the child process but not both, or that you switch to using GCD or
pthreads rather than forking to create parallelism.

Please also see the following in the fork(2) manpage:

CAVEATS
     There are limits to what you can do in the child process.  To be
totally safe you should restrict your-
     self to only executing async-signal safe operations until such
time as one of the exec functions is
     called.  All APIs, including global data symbols, in any
framework or library should be assumed to be
     unsafe after a fork() unless explicitly documented to be safe or
async-signal safe.  If you need to use
     these frameworks in the child process, you must exec.  In this
situation it is reasonable to exec your-
     self.
"""

cheers,
David


More information about the NumPy-Discussion mailing list