[SciPy-dev] problems with numpy.setuptools...
Sun Sep 30 00:10:57 CDT 2007
David M. Cooke wrote:
> When you mention the 'f2c' library, you mean the Fortran intrinsic and
> runtime libraries? We try to use the Fortran compiler for linking, so
> knowing those tends not to be needed.
But what if we want to use C++ code in the future ? Then you have no
choice but using the C++ compiler. But more to the point, this was an
example. All AC_F77_* macro (or AC_C_*, etc...) are much, much more
solid than distutils. And this is by design.
> For comparision, we've got 2348 lines of source in
> numpy/distutils/fcompiler/, and fortran.m4 included with autoconf is
> 1234 lines.
To make it clear: I was not talking about using autoconf (which for once
does not work on windows), but about the concept that almost everybody
agrees autoconf is good at, that is working on a new platform not
foreseen by the developer. And this is due to a fundamentally different
way to look for platform capabilities.
We have a huge advantage compared to autoconf: python vs m4 + shell.
Most of the lines in fortran.m4 are difficult because autoconf has to
deal with weird shells, antic platform bugs, and because of m4 and its
antic quoting rules.
> However, fortran.m4 mostly only deals with finding a Fortran
> compiler, finding flags for the libraries mentioned above, and name
> mangling (and nothing about optimisation flags). SCons also does its
> own handling of individual tools in much the same way as
> numpy.distutils (with, again, no optimisations).
What do you mean by no optimizations ? Dealing with facilities provided
by scons and dealing with capabilities are two different things, IMHO,
because I trust scons enough to work on what is supported. But for
example, when distutils is looking for something, it is always
hardcoded: looking for a library ? Distutils looks for a file, autoconf
is looking for something usable by the linker. This makes a huge
difference concerning the robustness (and the maintenance).
> Now, one design point that I think distutils gets wrong is to use
> lists for describing compiler options: the description of a compiler
> (or any of the command line tools used) should be an object, that
> 'knows' how to turn, say, a .c file into a .o file (something like
> scons' Builders, I think). We have some tools where the pattern
> EXECUTABLE_NAME + SOME_OPTIONS + file.c + MORE_OPTIONS + file.o +
> EVEN_MORE_OPTIONS doesn't work (most notably, linking shared libraries
> on AIX must be handled specially). The CCompiler and FCompiler classes
> are too high-level for this, as they contain knowledge about multiple
This is something that scons seems to make right.
> Fiddling compiler options shouldn't be too hard, espicially for the
> Fortran compilers; it's usually pretty obvious where to fiddle for
> those. But other things (C compilers, linking), I agree, could be
The point is, did it ever happen that somebody who had a problem with a
compiler flag solved it by himself ?
> Could you detail your problems? system_info.py needs to be cleaned up
> (removal of repeative code, for instance) and finding libraries
> standardised (for instance, there should always be an environment
> variable and/or distutils key for things system_info wants to find,
> that will override system_info).
system_info is looking for files, which is not the right way in my
opinion. For example, in audiolab, my system_info subclass is 70 lines
all this to find one library ! In scons (or autoconf, or any other
build tool I know), this is one line. Maybe I am doing things wrong, the
code started when I knew nothing about python and just copied what I
found in system_info.py, dunno. But the code in system_info does not
seem to imply this is simple or robust.
This is really striking for me: I have no problem with complicated code,
if it matches the complexity of the task it is trying to solve. But to
me, distutils makes anything trivial complicated (again, just talking
about the things related to compiled code).
> Blech, I hate the fact that scons doesn't respect my PATH. That means
> the compiler it may find is not necessarily the one that I would find
> at my command line. Makes bugs more obscure.
I fundamentally disagree on this one. But anyway, we don't even need to
agree, since you can easily import the variables you want. So we can
make this an option (which has always been my intention anyway), but by
default, not importing anything is more robust for deployment, no ?
People who have non standard tools or tools in non standard locations
are the ones who should make the effort, not the contrary IMHO.
> Not if I steal stuff from scons ;-)
I doubt you can "steal" things from scons, for the very reason you
> (Or, for that matter, if I actually *had* any time to hack on it,
> which I don't right now.)
> Speaking of which, if the scons developers would make their code an
> actual Python package (for one thing, install it by default in
> site-packages/, and try to keep interfaces constantish), I would
> support using scons modules in numpy.distutils. From the looks of it,
> it's nicely laid out, with a good set of abstractions. Instead,
> they've set it up so that scons must be in control, through SConstruct
I think this is partly due to the fact that scons support any python
starting from 1.5.2, and partly to the fact that scons is not imperative
(in fact, this is one key difference with waf AFAIK). But scons has a
package scons-local, which can be used as a drop in (in fact, that's
what I did in numpy.scons branch: in theory, we could make our
modification there), which somewhat alleviates the problem.
> Those have always had a faux-python feel to them for me: there
> are names injected into them (i.e. Environment) that don't come from
> an import statement, and the sharing of variables between SConscript
> files using the Export function is quite unpythonic.
I would agree in theory, but compared to python distutils, which one do
you think people can understand more easily ? For once, distutils (the
one from python) adds many members attributes at runtime, in many
functions (not just __init__); this combined to the fact that it is
mostly not commented + not documented + not maintained (nobody to ask
this, nothing on the internet), makes it harder for me to understand
than I ever did with the autotools, which are not known for their user
> Those are the big three. If it builds on those, we're probably 80% of
> the way there.
> Of course, it's the other 20% that'll take 80% of the work, as usual.
I agree that making the 20 % will take the most time (anybody with at
least some experience with cross platform building would, I guess). But
I think this will be much easier than with distutils.
> We can see what we can do to make it easier to get at those
> properties. For instance, it would be possible to separate the Fortran
> compiler detection/option-setting more from the distutils parts.
For example, one thing which caused me problems is to get the name of
the compiler: for unix compiler, just using the member compiler gives
me the executable, from which I have a dict to convert to scons names.
But MVSCCompiler does not have such a member; and to find the path of
the executable, *sometimes*, you need to call initialize (which of
course is not documented). This is not just unpythonic, this is just
really bad code; what's the point to have a base class for compiler if
this does not even have a common way to get the compiler name ?
>> To start building numpy itself with it would require more work (mostly
>> tests for fortran / C abi), but with the help of people willing to help,
>> I don't see major problems on this side, since scons provide a framework
>> for autoconf-like testing. Also, this will certainly use some facilities
>> of numpy.distutils (basically, for a first prototype, scons would mostly
>> replace system_info.py, command/build_clib.py, command/build_ext.py).
> If you have the time, I think it's worth a try. Unfortunately, I don't
> have much time to contribute to it.
> Basically, my opinion is not to replace numpy.distutils with scons
> (for the simple reason of interacting nicely with distutils).
Again, never had the intention to totally replace distutils: distutils
would still drive the whole build. Ideally, if it works out, I intend to
replace the build_ext and build_clib commands; it would still works by
calling setup.py build, it would still put everything in build, and
install using --prefix, etc... This would be totally transparent to the
user (except maybe for setting compiler at command line, and even, this
may be doable to stay 100% backward compatible at this level). You can
take a look at openalea, who did the exact same thing:
Everything else seems to work quite well in distutils, no ?
More information about the Scipy-dev