[Numpy-discussion] Combined versus separate build
Wed Jun 27 15:05:43 CDT 2012
On Wed, Jun 27, 2012 at 8:53 PM, Nathaniel Smith <email@example.com> wrote:
> On Wed, Jun 27, 2012 at 8:29 PM, David Cournapeau <firstname.lastname@example.org> wrote:
>> On Wed, Jun 27, 2012 at 8:07 PM, Nathaniel Smith <email@example.com> wrote:
>>> On Wed, Jun 27, 2012 at 7:50 PM, David Cournapeau <firstname.lastname@example.org> wrote:
>>>> On Wed, Jun 27, 2012 at 7:17 PM, Nathaniel Smith <email@example.com> wrote:
>>>>> Currently the numpy build system(s) support two ways of building
>>>>> numpy: either by compiling a giant concatenated C file, or by the more
>>>>> conventional route of first compiling each .c file to a .o file, and
>>>>> then linking those together. I gather from comments in the source code
>>>>> that the former is the traditional method, and the latter is the newer
>>>>> "experimental" approach.
>>>>> It's easy to break one of these builds without breaking the other (I
>>>>> just did this with the NA branch, and David had to clean up after me),
>>>>> and I don't see what value we really get from having both options --
>>>>> it seems to just double the size of the test matrix without adding
>>>> There is unfortunately a big value in it: there is no standard way in
>>>> C to share symbols within a library without polluting the whole
>>>> process namespace, except on windows where the default is to export
>>>> Most compilers support it (I actually know of none that does not
>>>> support it in some way or the others), but that's platform-specific.
>>> IIRC this isn't too tricky to arrange for with gcc
>> No, which is why this is supported for gcc and windows :)
>>>, but why is this an
>>> issue in the first place for a Python extension module? Extension
>>> modules are opened without RTLD_GLOBAL, which means that they *never*
>>> export any symbols. At least, that's how it should work on Linux and
>>> most Unix-alikes; I don't know much about OS X's linker, except that
>>> it's unusual in other ways.
>> The pragmatic answer is that if it were not an issue, python itself
>> would not bother with it. Every single extension module in python
>> itself is built from a single compilation unit. This is also why we
>> have this awful system to export the numpy C API with array of
>> function pointers instead of simply exporting things in a standard
> The array-of-function-pointers is solving the opposite problem, of
> exporting functions *without* having global symbols.
I meant that the lack of standard around symbols and namespaces is why
we have to do those hacks. Most platforms have much better solutions
to those problems.
>> See this: http://docs.python.org/release/2.5.3/ext/using-cobjects.html
>> Looking quickly at the 2.7.3 sources, the more detailed answer is that
>> python actually does not use RTLD_LOCAL (nor RTLD_GLOBAL), and what
>> happens when neither of them is used is implementation-dependent. It
>> seems to be RTLD_LOCAL on linux, and RTLD_GLOBAL on mac os x. There
>> also may be consequences on the use of RTLD_LOCAL in embedded mode (I
>> have ancient and bad memories with matlab related to this, but I
>> forgot the details).
> See, I knew OS X was quirky :-). That's what I get for trusting dlopen(3).
> But seriously, what compilers do we support that don't have
> -fvisibility=hidden? ...Is there even a list of compilers we support
> available anywhere?
Well, I am not sure how all this is handled on the big guys (bluegen
and co), for once.
There is also the issue of the consequence on statically linking numpy
to python: I don't what they are (I would actually like to make
statically linked numpy into python easier, not harder).
More information about the NumPy-Discussion