[Numpy-discussion] [Announce] numpy.scons , ALPHA version
Fri Oct 26 07:20:34 CDT 2007
I've finally managed to implement most of the things I wanted for
numpy.scons, hence a first alpha. Compared to the 2d milestone from a
few days ago, a few optimized libraries are supported (ATLAS, Sun
sunperf, Apple Accelerate and vecLib, Intel MKL).
Outside people interested in numpy.scons, people who have trouble
building numpy may want to try this. In particular, using MKL or sunperf
should be easier (for sunperf, it should work out of the box if sun
compilers are used). Also, compiling with intel compilers on linux
should be easier.
DO NOT USE THIS FOR PRODUCTION USE ! I am interested in success (that is
build and all the test pass) as well as failure.
Numpy developers: I would be interested in comments about the code:
- the package are built with the setupscons.py/SConstruct
- most of the new code is in numpy/distutils/scons
How to test
Grab the sources:
svn co http://svn.scipy.org/svn/numpy/branches/numpy.scons
And then just do: python setup.py scons; python setup.py install. If you
have multiple CPU/Cores, you may want to use the parallel build options:
python setup.py scons --jobs=N with N an integer (number of CPU/CORES +
1 is a rule of thumb).
For non default compilers, use the --compiler and --fcompiler options
For the MKL, you should add a mkl option in the site.cfg. For ATLAS or
sunperf, if it is not in standard places, it won't work (yet).
The tested platforms have been tested:
- linux (ubuntu x86) with gcc, g77 and ATLAS.
- linux (x86 and x64) with gcc, without BLAS or LAPACK.
- linux (ubuntu x86) with gcc, g77, and mkl (9.1.023).
- linux (ubuntu x86) with intel compilers.
- windows (32 bits) with mingw.
- windows (32 bits) with VS 2003.
- solaris (solaris studio express 12) with sunperf on x86.
- Mac Os X Intel with Accelerate framework.
- People reported basic success on IRIX, too.
What does NOT work
One important target missing is windows 64, but this should not be too
difficult to solve.
There are still many corner cases not yet solved (in particular some
windows things, most libraries path cannot yet be overriden in
site.cfg); also, I do not attempt to be 100 % backward compatible with
the current build system (that for a given environment, you won't
necessarily have the same build options). Some things are not yet
supported either (overriding libraries with env variables, for example,
is not supported).
There is no real doc yet (I will do this once the basic public API is
The main difference compared to the second milestone is the support for
libraries checkers (think BLAS, Atlas, etc...). There is now support to
check for different libraries, with different compilers. This
effectively replaces numpy.distutils.system_info. The way I've
implemented those checks are fundamentally different compared to the
distutils way: instead of looking for files, I look for capabilities at
For example, for mkl, instead of looking for libmkl.so, etc... I try to
compile, link and run a small mkl program: if any of the step failed,
then the mkl is not used. This means that I can detect config errors
much sooner; also, since I use rpath, there is no need to set things
like LD_LIBRARY_PATH and so on. If the path is set right in site.cfg,
then everything is taken care of.
Also, as before, a general library checker for new libraries is
available: look at basic examples in numpy/scons_fake/. In particular,
there is a NumpyCheckLib which can be used to check a library with some
symbols, whose paths can be overriden with a site.cfg.
The other big work has been on fortran support. Here too, the way I
implemented it is different than numpy.distutils. Instead of harcoding
flags depending on versions, I use code snipets, to get link options,
mangling, and so on. For example, if you call the following in a Sconscript:
This will try to get the mangling of the F77 compiler, and if
sucessfull, will set a python function mangle, which returns the fortran
name from the 'common' name. For example, for g77, mangle('dgemm') will
returns dgemm_. Since the mangling is found at runtime, this can be used
for not yet known compilers.
This was quite a lot of work, and I wanted to get something working as
fast as possible. As such, the code is not always clean. Once I am
satisfied with the global 'feel' of the new build, I will sanitize the
API and implement the doc. Still, I believe the code to be much easier,
simpler and robust than distutils (everything, from fortran support to
libraries checkers is less than 2000 lines of python code, including
tests; of course, this is largely thanks to scons).
The thing I am the less sure about is how to combine checkers in
meta-checkers: for example, I have checker for mkl, ATLAS, sunperf, and
we sometimes want a BLAS checker, which would check all those as long as
nothing is found. API-wise, I am not yet satisfied with the current way.
I also want to play nice with f2py and scipy (totally untested for now).
More information about the Numpy-discussion