[Numpy-discussion] Combined versus separate build

David Cournapeau cournape@gmail....
Wed Jun 27 14:29:54 CDT 2012


On Wed, Jun 27, 2012 at 8:07 PM, Nathaniel Smith <njs@pobox.com> wrote:
> On Wed, Jun 27, 2012 at 7:50 PM, David Cournapeau <cournape@gmail.com> wrote:
>> On Wed, Jun 27, 2012 at 7:17 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>> Currently the numpy build system(s) support two ways of building
>>> numpy: either by compiling a giant concatenated C file, or by the more
>>> conventional route of first compiling each .c file to a .o file, and
>>> then linking those together. I gather from comments in the source code
>>> that the former is the traditional method, and the latter is the newer
>>> "experimental" approach.
>>>
>>> It's easy to break one of these builds without breaking the other (I
>>> just did this with the NA branch, and David had to clean up after me),
>>> and I don't see what value we really get from having both options --
>>> it seems to just double the size of the test matrix without adding
>>> value.
>>
>> There is unfortunately a big value in it: there is no standard way in
>> C to share symbols within a library without polluting the whole
>> process namespace, except on windows where the default is to export
>> nothing.
>>
>> Most compilers support it (I actually know of none that does not
>> support it in some way or the others), but that's platform-specific.
>
> IIRC this isn't too tricky to arrange for with gcc

No, which is why this is supported for gcc and windows :)

>, but why is this an
> issue in the first place for a Python extension module? Extension
> modules are opened without RTLD_GLOBAL, which means that they *never*
> export any symbols. At least, that's how it should work on Linux and
> most Unix-alikes; I don't know much about OS X's linker, except that
> it's unusual in other ways.

The pragmatic answer is that if it were not an issue, python itself
would not bother with it. Every single extension module in python
itself is built from a single compilation unit. This is also why we
have this awful system to export the numpy C API with array of
function pointers instead of simply exporting things in a standard
way.

See this: http://docs.python.org/release/2.5.3/ext/using-cobjects.html

Looking quickly at the 2.7.3 sources, the more detailed answer is that
python actually does not use RTLD_LOCAL (nor RTLD_GLOBAL), and what
happens when neither of them is used is implementation-dependent. It
seems to be RTLD_LOCAL on linux, and RTLD_GLOBAL on mac os x. There
also may be consequences on the use of RTLD_LOCAL in embedded mode (I
have ancient and bad memories with matlab related to this, but I
forgot the details).

David


More information about the NumPy-Discussion mailing list