[Numpy-discussion] Bitting the bullet: using scons to build extensions inside distutils ?

David Cournapeau cournape@gmail....
Sat Sep 15 15:59:49 CDT 2007


On 9/16/07, Pearu Peterson <pearu@cens.ioc.ee> wrote:
> On Sat, September 15, 2007 9:41 pm, David Cournapeau wrote:
>
> >    Starting thinking over the whole distutils thing, I was thinking
> > what people would think about using scons inside distutils to build
> > extension.
>
> Are you thinking of Python distutils or numpy.distutils?
> In the case of Python distutils, then I think this is a
> wrong list to discuss this subject.
>
First, let's get one thing straight: this is not about replacing
distutils, but about using scons for what it is good at (building
code), instead of extending distutils for what it is was not conceived
for in the first place (building compiled code in a flexible way).

I certainly do not intend to replace distutils, but to use scons a a
build tool for extensions (that is, scons would not be aware of python
code, but would be used to compile all the extensions; for pure python
code, I don't think there are major issues related to distutils ?).

> > The more I think about it, the more I think than distutils
> > not being maintained,
>
> numpy.distutils is being maintained.
I was speaking about python distutils, of course.

> > and numpy/scipy building needs being much more
> > complicated (at least different) than usual python extension, trying
> > to circumvent distutils problems is an ever ending fight.
>
> why would be scons be different when new architectures and platforms
> that need to be supported are developed now and in future?

scons already knows about many platforms/compiler combinations: except
fortran, I think scons supports more than numpy.distutils now. And it
is much less fragile: you can extend it much more easily, which is a
key point IMHO.

>
> > Scons, being
> > developped as a Make replacement, can do all we would like to be able
> > to do with distutils, including:
> >  - building shared or static libraries, with dependencies (knows how
> > to do it on many platforms).
> >  - can build each object file independently (e.g different compiler
> > options)
>
> IMHO, this is the only important feature that distutils is missing
> and would be difficult to add to numpy.distutils because of distutils
> design.

Whereas it would be trivial with scons :)

> >  - have basic facility to look for libraries (ala autoconf. By basic,
> > I mean it is far from being as complete as autoconf, but is much
> > better than distutils).
>
> Can it achive the same as numpy/distutils/system_info.py?

I think so. Finding functions, headers, types, is available, and you
have tools like TryBuild, TryLink and so on which make customizing our
own really easy. That is, instead of digging into undocumented python
distutils api, you have basic, abstracted tools, on which you can
build more high level things.
>
> > Scons has also the following advantages:
> >  - written in python, can be distributed with numpy (by this, I mean
> > AFAIK, license-wise, it is ok, and its size is not big): does not add
> > additional dependency.
> >  - can be called within distutils quite easily.
>
> Scons does not have Fortran compiler support, at least not as advanced
> as numpy.distutils.fcompiler provides, afaik. I think this is the main
> reason why numpy.distutils cannot be replaced with scons without
> having much of effort adding this feature to scons.

I don't know much about fortran, so what are the specific needs for
numpy ? scons has a fortran builder, and can be customized. When I
take a look at the fcompiler module in numpy.distutils, I get the
feeling that most of the hard work would be unnecessary for scons, no
?

>
> I think the idea of replacing distutils with some other tool which
> would be easier to use and to extend has brought up every one or two years.
> I think it hasn't happened because it would require not just some
> work but lots of it. Whatever people say about distutils, it is still
> a good tool that has lots of know-how in it for doing its job in a large
> variety of platforms we have today.
I agree, and I do not believe much is throwing code away. Here, I am
merely suggesting to extend distutils using scons for extension.
Actually, some people got the idea of what I have in mind before me:

http://openalea.gforge.inria.fr/dokuwiki/doku.php?id=packages:compilation_installation:deploy:deploy

(I didn't look too much at the code, because this is GPL, and we
cannot just copy it without infringing copyright; I just checked it
was doing what I thought it was doing).

To sum up: keep distutils for what it is not too bad at, and use scons
instead of extending distutils for what it was not conceived for and
where scons shines.

David


More information about the Numpy-discussion mailing list