[SciPy-dev] problems with numpy.setuptools...
Fri Sep 28 23:43:27 CDT 2007
Pearu Peterson wrote:
> On Fri, September 28, 2007 11:40 pm, Stefan van der Walt wrote:
>> I would prefer if numpy were *always* in a release-ready state.
> Most of the time it actually is.
>> Why can't we instrument tests for distutils? If the code is so confusing
>> that we can't test it (or "practical corner cases"), should we be
>> using it as a core ingredient in the first place?
> The code is not so confusing - it has well-defined (though undocumented)
> structure that can be extended after one gets an idea how things
> work in distutils.
Before I start to make my points, and because I feel this discussion is
heating a bit, let me say that I never intend to criticize any numpy's
developers work/code. I understand that distutils is complicated to
extend, that numpy's needs go far beyond the usual needs of python
package, and I really think that numpy.distutils has already achieved
quite a lot. All other impressions are surely du to the fact that
English is not my native language :)
Now, although I certainly do not have as much experience as you with
distutils, I have used autotools for non trivial projects, I have used
scons as well, and I have some experience with complicated gnu make
based projects (eg my numpy.gar script which can builds numpy and all
its dependencies in different configuration/compilers on linux/unix).
This is not to say I am knowledgeable, but at least I think I am aware
of the difficulties of a cross platform build system.
> The variety of different compilers and platforms makes developing
> distutils difficult. I guess none of the developers who dare to
> touch distutils have access to all platforms and compilers that
> we are trying to support. This fact will not change after switching
> to some other tool such as scons.
That's where I think we fundamentally disagree. I for one think
distutils (here I mean the official, python.distutils, and only to build
C/Fortran extensions; I will never talk about any other capabilities of
distutils) is fundamentally flawed wrt to several key points, and I am
not even talking about its implementation. Since you are much more
familiar with distutils than me, I would really appreciate being told wrong:
1 - the way it tries to detect the platform capability. When you
try to adapt to a platform in a build tool, you have at least two
possibilities: either you hardcode every new platform, or you try to get
the information from the platform. Distutils does the former, autoconf
does the latter. For me, there is absolutely no question which one is
better in this respect. This alone explains a lot of distutils fragility
IMHO. For example, numpy.distutils define the "f2c" library for each new
tool, but autoconf (more explicitely, the autoconf macro
AC_F77_LIBRARY_LDFLAGS) finds it automatically from compiler output.
This is a fundamental point, maybe the only most important one.
2 - numpy.distutils is difficult to extend. This, I think we all
agree. This is also significant, because it means that when
numpy.distutils fails, only a few people can change the situation. This
should not be the case. If the problem is one compiler flag, it should
not require anyone's involvement but the ones who experience the problem.
3 - Customizing compilation options: this is extremely difficult
right now, and this again by design.
4 - Detecting libraries is extremely difficult: the first time I
used system_info, it was totally wrong, and I had to modify it for each
new platform. This is not good: I needed more time to make it right than
I ever did for autotools based projects of mine, which really says a lot
in my book.
5 - By design, scons does not depend on the environment. That is, it
does NOT use PATH information, LD_LIBRARY_FLAGS, etc... which again
makes it much more reliable. Of course, you can add command line
arguments for configuration, but by default, nothing will stab you in
Now, you would rightly argue that doing all this in scons/whatever would
take a lot of time just to replicate distutils, and you would be right.
But I think it would require *less* time than implementing the things
stated eg in http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp.
But I don't ask you to take my word on it: I am more than willing to
code a prototype of what I have in mind. But then, I need to know what
is required for a prototype, to make a plan of what needs to be implemented:
- Is a prototype required to support all platforms supported now ?
- Is a prototype which can build numpy on Mac OS X/Linux/Win32
enough (keeping in mind that the key point is that adding new platforms
would be much easier once the prototype is done). In which configuration ?
- If required, I could try to make it possible to have two top setup
scripts, one which uses the current distutils, one which use the
As said earlier, I have started hacking on a numpy branch which, right now:
- implements a scons command
- a support library, which enables to build/run ctypes extensions
working on many platforms (all the one I managed to use on the buildbot)
- a support library to find libraries and headers in standard
location on all supported platforms by scons (of which numpy.distutils
is a subset AFAIK), and can be customized using site.cfg.
One problem I did not foresee is how difficult it is to be sure that
scons uses exactly the same tools than distutils (because distutils has
no common api to get things such as compiler, compiler path, etc...);
that's why contrary to what I thought first, using scons to build all
the compiled extension may actually be easier than making them coexist,
because scons and distutils have fundamentally different ways of looking
To start building numpy itself with it would require more work (mostly
tests for fortran / C abi), but with the help of people willing to help,
I don't see major problems on this side, since scons provide a framework
for autoconf-like testing. Also, this will certainly use some facilities
of numpy.distutils (basically, for a first prototype, scons would mostly
replace system_info.py, command/build_clib.py, command/build_ext.py).
Does it sound like an acceptable plan to you ?
More information about the Scipy-dev