[Numpy-discussion] How many build systems do we need?

Nathaniel Smith njs@pobox....
Sun Jan 13 11:11:47 CST 2013

On Sun, Jan 13, 2013 at 3:47 PM, Charles R Harris
<charlesr.harris@gmail.com> wrote:
> On Sun, Jan 13, 2013 at 7:30 AM, Nathaniel Smith <njs@pobox.com> wrote:
>> On Sun, Jan 13, 2013 at 5:34 AM, Charles R Harris
>> <charlesr.harris@gmail.com> wrote:
>> > Hi All,
>> >
>> > In the continuing proposal for cleanups, note that we currently support
>> > three (3!) build systems, distutils, scons, and bento. That's a bit much
>> > to
>> > maintain when contemplating changes, and scons and bento both have
>> > external
>> > dependencies. Can we dispense with any of these? Thoughts?
>> I think it's actually 6 build systems, because each build system
>> supports two modes: compiling each source file separately before
>> linking, and concatenating everything into one big file and compiling
>> that.
>> It's been proposed that we phase out the one-file build (which is
>> currently the default):
>>   http://mail.scipy.org/pipermail/numpy-discussion/2012-June/063015.html
>>   https://github.com/numpy/numpy/issues/315
>> The separate compilation approach is superior in every way, so long as
>> it works. There is a theory that on some system somewhere there might
>> be a broken compiler/linker which make it not work[1], but we don't
>> actually know of any such system.
>> Shall we switch the default to separate compilation for 1.8 and see if
>> anyone notices?
> +1


>> [1] The problem is that we need to make sure that symbols defined in
>> numpy .c files are visible to other numpy .c files, but not to
>> non-numpy code linked into the same process; this is a problem that
>> the C standard didn't consider, so it requires system-specific
>> fiddling. However that fiddling is pretty standard these days.
> And do we really care? I've compiled and statically linked libraries using
> setup.py because it is more easily portable than make, and on windows few
> symbols are exposed by default while on linux most are, but who looks?
> Exposing unneeded symbols is  a bit of a wart but I don't think it matters
> that much for most things.

I guess it's like many things in programming -- it doesn't matter
until it does. (And then you realize that you should have done things
properly from the start because you have a horrible hairball of "eh,
does it really matter?" to sort out.)

OTOH it's easy to build a static object without unneeded symbols, at
least if you have access to gnu ld. I assume that most of these
systems that don't have dynamic linkers still use some variant of

# Let's take the .o files from separate compilation and make a single
.o file suitable for static linking
$ NPY_SEPARATE_COMPILATION=1 python setup.py build
$ cd build/temp.*/numpy/core/src/multiarray
# Combine all .o files into a single .o file, resolving all internal symbols:
$ ld -r *.o -o multiarray-all.o
# This file still exports a ton of junk...
$ nm multiarray-all.o | wc -l
# But now, we can strip out all the stuff we don't want to be public
$ strip --strip-all --keep-symbol initmultiarray multiarray-all.o
# Ta-da, a single-file static Python module that exports only the
module setup symbol:
$ nm multiarray-all.o
0000000000047a40 T initmultiarray


More information about the NumPy-Discussion mailing list