[Numpy-discussion] Combined versus separate build

Nathaniel Smith njs@pobox....
Sun Jul 1 14:32:41 CDT 2012

On Sun, Jul 1, 2012 at 7:36 PM, David Cournapeau <cournape@gmail.com> wrote:
> On Sun, Jul 1, 2012 at 6:36 PM, Nathaniel Smith <njs@pobox.com> wrote:
>> On Wed, Jun 27, 2012 at 9:05 PM, David Cournapeau <cournape@gmail.com> wrote:
>>> On Wed, Jun 27, 2012 at 8:53 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>>> But seriously, what compilers do we support that don't have
>>>> -fvisibility=hidden? ...Is there even a list of compilers we support
>>>> available anywhere?
>>> Well, I am not sure how all this is handled on the big guys (bluegen
>>> and co), for once.
>>> There is also the issue of the consequence on statically linking numpy
>>> to python: I don't what they are (I would actually like to make
>>> statically linked numpy into python easier, not harder).
>> All the docs I can find in a quick google seem to say that bluegene
>> doesn't do shared libraries at all, though those may be out of date.
>> Also, it looks like our current approach is not doing a great job of
>> avoiding symbol table pollution... despite all the NPY_NO_EXPORTS all
>> over the source, I still count ~170 exported symbols on Linux with
>> numpy 1.6, many of them with non-namespaced names
>> ("_n_to_n_data_copy", "_next", "npy_tan", etc.) Of course this is
>> fixable, but it's interesting that no-one has noticed. (Current master
>> brings this up to ~300 exported symbols.)
>> It sounds like as far as our "officially supported" platforms go
>> (linux/windows/osx with gcc/msvc), then the ideal approach would be to
>> use -fvisibility=hidden or --retain-symbols-file to convince gcc to
>> hide symbols by default, like msvc does. That would let us remove
>> cruft from the source code, produce a more reliable result, and let us
>> use the more convenient separate build, with no real downsides.
> What cruft would it allow us to remove ? Whatever method we use, we
> need a whitelist of symbols to export.

No, right now we don't have a whitelist, we have a blacklist -- every
time we add a new function or global variable, we have to remember to
add a NPY_NO_EXPORT tag to its definition. Except the evidence says
that we don't do that reliably. (Everyone always sucks at maintaining
blacklists, that's the nature of blacklists.) I'm saying that we'd
better off if we did have a whitelist. Especially since CPython API
makes maintaining this whitelist so very trivial -- each module
exports exactly one symbol!

> On the exported list I see on mac, most of them are either from
> npymath (npy prefix) or npysort (no prefix, I think this should be
> added). Once those are ignored as they should be, there are < 30
> symbols exported.
>> (Static linking is trickier because no-one uses it anymore so the docs
>> aren't great, but I think on Linux at least you could accomplish the
>> equivalent by building the static library with 'ld -r ... -o
>> tmp-multiarray.a; objcopy --keep-global-symbol=initmultiarray
>> tmp-multiarray.a multiarray.a'.)
> I am not sure why you say that static linking is not used anymore: I
> have met some people who do statically link numpy into python.

Yes, of course, or I wouldn't have bothered researching it. But this
research would have been easier if there were enough of a user base
that the tools makers actually paid any attention to supporting this
use case, is all I was saying :-).

>> Of course there are presumably other platforms that we don't support
>> or test on, but where we have users anyway. Building on such a
>> platform sort of intrinsically requires build system hacks, and some
>> equivalent to the above may well be available (e.g. I know icc
>> supports -fvisibility). So I while I'm not going to do anything about
>> this myself in the near future, I'd argue that it would be a good idea
>> to:
>>  - Switch the build-system to export nothing by default when using
>> gcc, using -fvisibility=hidden
>>  - Switch the default build to "separate"
>>  - Leave in the single-file build, but not "officially supported",
>> i.e., we're happy to get patches but it's not used on any systems that
>> we can actually test ourselves. (I suspect it's less fragile than the
>> separate build anyway, since name clashes are less common than
>> forgotten include files.)
> I am fine with making the separate build the default (I have a patch
> somewhere that does that on supported platforms), but not with using
> -fvisibility=hidden. When I implemented the initial support around
> this, fvisibility was buggy on some platforms, including mingw 3.x

It's true that mingw doesn't support -fvisibility=hidden, but that's
because it would be a no-op; windows already works that way by

> I don't think changing what our implementation does here is worthwhile
> given that it works, and fsibility=hidden has no big advantages (you
> would still need to mark the functions to be exported).

But there are only about 10 functions that we need to export, and that
list never changes; OTOH there are tons and tons of functions that we
want to *not* export, and that list changes constantly.


More information about the NumPy-Discussion mailing list