[Numpy-discussion] Combined versus separate build

David Cournapeau cournape@gmail....
Wed Jun 27 15:07:13 CDT 2012


On Wed, Jun 27, 2012 at 8:57 PM, Dag Sverre Seljebotn
<d.s.seljebotn@astro.uio.no> wrote:
> On 06/27/2012 09:53 PM, Nathaniel Smith wrote:
>> On Wed, Jun 27, 2012 at 8:29 PM, David Cournapeau<cournape@gmail.com>  wrote:
>>> On Wed, Jun 27, 2012 at 8:07 PM, Nathaniel Smith<njs@pobox.com>  wrote:
>>>> On Wed, Jun 27, 2012 at 7:50 PM, David Cournapeau<cournape@gmail.com>  wrote:
>>>>> On Wed, Jun 27, 2012 at 7:17 PM, Nathaniel Smith<njs@pobox.com>  wrote:
>>>>>> Currently the numpy build system(s) support two ways of building
>>>>>> numpy: either by compiling a giant concatenated C file, or by the more
>>>>>> conventional route of first compiling each .c file to a .o file, and
>>>>>> then linking those together. I gather from comments in the source code
>>>>>> that the former is the traditional method, and the latter is the newer
>>>>>> "experimental" approach.
>>>>>>
>>>>>> It's easy to break one of these builds without breaking the other (I
>>>>>> just did this with the NA branch, and David had to clean up after me),
>>>>>> and I don't see what value we really get from having both options --
>>>>>> it seems to just double the size of the test matrix without adding
>>>>>> value.
>>>>>
>>>>> There is unfortunately a big value in it: there is no standard way in
>>>>> C to share symbols within a library without polluting the whole
>>>>> process namespace, except on windows where the default is to export
>>>>> nothing.
>>>>>
>>>>> Most compilers support it (I actually know of none that does not
>>>>> support it in some way or the others), but that's platform-specific.
>>>>
>>>> IIRC this isn't too tricky to arrange for with gcc
>>>
>>> No, which is why this is supported for gcc and windows :)
>>>
>>>> , but why is this an
>>>> issue in the first place for a Python extension module? Extension
>>>> modules are opened without RTLD_GLOBAL, which means that they *never*
>>>> export any symbols. At least, that's how it should work on Linux and
>>>> most Unix-alikes; I don't know much about OS X's linker, except that
>>>> it's unusual in other ways.
>>>
>>> The pragmatic answer is that if it were not an issue, python itself
>>> would not bother with it. Every single extension module in python
>>> itself is built from a single compilation unit. This is also why we
>>> have this awful system to export the numpy C API with array of
>>> function pointers instead of simply exporting things in a standard
>>> way.
>>
>> The array-of-function-pointers is solving the opposite problem, of
>> exporting functions *without* having global symbols.
>>
>>> See this: http://docs.python.org/release/2.5.3/ext/using-cobjects.html
>>>
>>> Looking quickly at the 2.7.3 sources, the more detailed answer is that
>>> python actually does not use RTLD_LOCAL (nor RTLD_GLOBAL), and what
>>> happens when neither of them is used is implementation-dependent. It
>>> seems to be RTLD_LOCAL on linux, and RTLD_GLOBAL on mac os x. There
>>> also may be consequences on the use of RTLD_LOCAL in embedded mode (I
>>> have ancient and bad memories with matlab related to this, but I
>>> forgot the details).
>>
>> See, I knew OS X was quirky :-). That's what I get for trusting dlopen(3).
>>
>> But seriously, what compilers do we support that don't have
>> -fvisibility=hidden? ...Is there even a list of compilers we support
>> available anywhere?
>
> You could at the very least switch the default for a couple of releases,
> introducing a new flag with a "please email numpy-discussion if you use
> this" note, and see if anybody complains?

Yes, we could. That's actually why I set up travis-CI to build both
configurations in the first place :) (see
https://github.com/numpy/numpy/issues/315)

David


More information about the NumPy-Discussion mailing list