[Numpy-discussion] Combined versus separate build

Nathaniel Smith njs@pobox....
Mon Jul 2 17:34:49 CDT 2012

On Sun, Jul 1, 2012 at 9:17 PM, David Cournapeau <cournape@gmail.com> wrote:
> On Sun, Jul 1, 2012 at 8:32 PM, Nathaniel Smith <njs@pobox.com> wrote:
>> On Sun, Jul 1, 2012 at 7:36 PM, David Cournapeau <cournape@gmail.com> wrote:
>>> On Sun, Jul 1, 2012 at 6:36 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>>> On Wed, Jun 27, 2012 at 9:05 PM, David Cournapeau <cournape@gmail.com> wrote:
>>>>> On Wed, Jun 27, 2012 at 8:53 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>>>>> But seriously, what compilers do we support that don't have
>>>>>> -fvisibility=hidden? ...Is there even a list of compilers we support
>>>>>> available anywhere?
>>>>> Well, I am not sure how all this is handled on the big guys (bluegen
>>>>> and co), for once.
>>>>> There is also the issue of the consequence on statically linking numpy
>>>>> to python: I don't what they are (I would actually like to make
>>>>> statically linked numpy into python easier, not harder).
>>>> All the docs I can find in a quick google seem to say that bluegene
>>>> doesn't do shared libraries at all, though those may be out of date.
>>>> Also, it looks like our current approach is not doing a great job of
>>>> avoiding symbol table pollution... despite all the NPY_NO_EXPORTS all
>>>> over the source, I still count ~170 exported symbols on Linux with
>>>> numpy 1.6, many of them with non-namespaced names
>>>> ("_n_to_n_data_copy", "_next", "npy_tan", etc.) Of course this is
>>>> fixable, but it's interesting that no-one has noticed. (Current master
>>>> brings this up to ~300 exported symbols.)
>>>> It sounds like as far as our "officially supported" platforms go
>>>> (linux/windows/osx with gcc/msvc), then the ideal approach would be to
>>>> use -fvisibility=hidden or --retain-symbols-file to convince gcc to
>>>> hide symbols by default, like msvc does. That would let us remove
>>>> cruft from the source code, produce a more reliable result, and let us
>>>> use the more convenient separate build, with no real downsides.
>>> What cruft would it allow us to remove ? Whatever method we use, we
>>> need a whitelist of symbols to export.
>> No, right now we don't have a whitelist, we have a blacklist -- every
>> time we add a new function or global variable, we have to remember to
>> add a NPY_NO_EXPORT tag to its definition. Except the evidence says
>> that we don't do that reliably. (Everyone always sucks at maintaining
>> blacklists, that's the nature of blacklists.) I'm saying that we'd
>> better off if we did have a whitelist. Especially since CPython API
>> makes maintaining this whitelist so very trivial -- each module
>> exports exactly one symbol!
> There may be some confusion on what NPY_NP_EXPORT does: it marks a
> function that can be used between compilation units but is not
> exported. The choice is between static and NPY_NO_EXPORT, not between
> NPY_NO_EXPORT and nothing. In that sense, marking something
> NPY_NO_EXPORT is a whitelist.
> If we were to use -fvisibility=hidden, we would still need to mark
> those functions static (as it would otherwise publish functions in the
> single file build).

To be clear, this subthread started with the caveat *as far as our
"officially supported" platforms go* -- I'm not saying that we should
go around and remove all the NPY_NO_EXPORT macros tomorrow.

However, the only reason they're actually needed is for supporting
platforms where you can't control symbol visibility from the linker,
and AFAICT we have no examples of such platforms to hand. So I'm
questioning the wisdom of maintaining multiple parallel build systems
etc. just for this hypothetical benefit.

>> Yes, of course, or I wouldn't have bothered researching it. But this
>> research would have been easier if there were enough of a user base
>> that the tools makers actually paid any attention to supporting this
>> use case, is all I was saying :-).
>>>> Of course there are presumably other platforms that we don't support
>>>> or test on, but where we have users anyway. Building on such a
>>>> platform sort of intrinsically requires build system hacks, and some
>>>> equivalent to the above may well be available (e.g. I know icc
>>>> supports -fvisibility). So I while I'm not going to do anything about
>>>> this myself in the near future, I'd argue that it would be a good idea
>>>> to:
>>>>  - Switch the build-system to export nothing by default when using
>>>> gcc, using -fvisibility=hidden
>>>>  - Switch the default build to "separate"
>>>>  - Leave in the single-file build, but not "officially supported",
>>>> i.e., we're happy to get patches but it's not used on any systems that
>>>> we can actually test ourselves. (I suspect it's less fragile than the
>>>> separate build anyway, since name clashes are less common than
>>>> forgotten include files.)
>>> I am fine with making the separate build the default (I have a patch
>>> somewhere that does that on supported platforms), but not with using
>>> -fvisibility=hidden. When I implemented the initial support around
>>> this, fvisibility was buggy on some platforms, including mingw 3.x
>> It's true that mingw doesn't support -fvisibility=hidden, but that's
>> because it would be a no-op; windows already works that way by
>> default...
> That's not my understanding: gcc behaves on windows as on linux (it
> would break too many softwares that are the usual target of mingw
> otherwise), but the -fvisibility flag is broken on gcc 3.x. The more
> recent mingw supposedly handle this better, but we can't use gcc 4.x
> because of another issue regarding private dll sharing :)

I don't have windows to test, but everyone else on the internet seems
to think mingw works the way I said, with __declspec and all... you
aren't thinking of cygwin, are you? (see e.g.


More information about the NumPy-Discussion mailing list