[Numpy-discussion] For review: first milestone of scons support in numpy
Fri Oct 12 03:33:53 CDT 2007
Pearu Peterson wrote:
> Yes. I was considering the situations where distutils in not used at
> all. May be it will be relevant in future..
If distutils is not used at all, there will be more difficult problems
to solve than this one, then. I think it is safe to assume that
distutils will stay there for quite some time, no ?
Note that I try hard in numpy/scons to separate the scons specific from
the parts which are not specific to scons. If in the future, we decide
to go on the waf route, or whatever, I hope that most of the logic would
be easily reused: for example, parsing the fortran link output to guess
the link options for C/Fortran is totally independant from scons (this
has the nice advantage of making regression tests possible on all
> I would prefer the former solution where each package is totally
> independent. Let's first see what is the acctual performance and then
> maybe consider more complex solutions if the build is really slow.
I measured the initialization time of scons if called for every
subpackage in numpy/: this adds less than two percents of total build
time, so this is not as bad as I feared (specially since only a few
package would need scons: pure python ones do not need them). And scons
can also give some speed advantage, because it explicitely support in a
totally safe way parallel builds.
So basically, I went to the same conclusion than you. I have started
porting numpy.core to scons (by far the most difficult subpackage, at
least a priori) following this route.
> These macros are fine. May be we need more such macros for F90 compilers
> but we can discuss about these when the corresponding issues will emerge.
Oh, I forgot to say it, but the macro would be implemented for both F77
and F90 (and F2003 or whatever should we need it). As most of the logic
is independent of the version used, this is not a problem.
> However, if you could also provide the macros
> that f2py uses then everything would work with no change in f2py part.
> f2py uses the following defintion of F_FUNC, for example:
I can of course change the name of the macro (this could be an option,
even). I guess it is better to follow f2py convention than autotools
here, in our case.
> system_info has a feature of providing architecture specific
> and optimization flags for fortran compilers. Could scons
> support such a feature?
scons uses the concept of Environment objects: they have sensible
defaults (compiler/platform combination), but you can override anything
you want, and they are used for compilation.
env = Environment()
env.AppendUnique(F77FLAGS = "-O3")
env.Program('hello', source = ['hello.f'])
Of course, in our case, we should provide some support such as instead
of adding -O3, we have something like a FULL_OPTIMIZED variable which
contains the optimization flags tweaked for a particular
compiler/platform. Better, we can check that a given flag is actually
supported. This is something I have not done yet, but that's the kind of
things scons does really well, so I don't see any problems here.
The main problem right now is to convert all the checks in
numpy/core/setup.py, which is the only non trivial setup.py file as far
as I can tell in numpy. This will take a few hours to get it right on
major platforms :)
P.S If you have access to some fortran compiler you would like to check,
you can check my bzr branch on launchpad (this is done outside numpy
because some things may be merged to scons sources at some point):
And test it by: scons CC=ccompiler F77=f77compiler F90=f90compiler. For
example: scons CC=icc F77=g77 F90=ifort.
(If you change compiler, you may need to add the option --config=force).
This is a "fake" sconscript which tests the following:
- does the fortran compiler works (e.g. can compile a dummy program)
- find the dummy main
- find the verbose flag (useful to find informations at runtime)
- find the name mangling
- find the necessary options for C/Fortran mix (only implemented for
F77 for now).
There is a pretty good chance that you will be able to break it (in
particular finding the options for C/Fortran: I have not yet implemented
all the necessary parsing), but in this case, please provides me with
the config.log (which logs all the configuration dones by scons).
More information about the Numpy-discussion