[Numpy-discussion] For review: first milestone of scons support in numpy

David Cournapeau david@ar.media.kyoto-u.ac...
Thu Oct 11 03:54:13 CDT 2007


Pearu Peterson wrote:
> Hi,
>
> Examples look good. It seems that you have lots of work ahead;) to add
> numpy.distutils features that are required to build numpy/scipy.
>
Hi Pearu,

    Thanks for reviewing this, especially since you are arguably the 
most knowledgeable about this part of numpy :)
> Few comments:
> 1) Why SConstruct does not have extension? It looks like a python file
> and .py extension could be used.
>
It looks like but isn't. This is actually the part of scons I like the 
least: scons uses python, but sconscripts files are not proper python  
modules. Sconscripts files are used declaratively:

http://www.scons.org/doc/HTML/scons-user/x348.html

I think this is one of the biggest design mistake of scons. Waf, which 
was briefly mentioned by David M. Cooke, and which is based on scons, 
does not do this, though (I did not use waf because it is new, is 
maintained only by one developer, and is more geared toward unix, that 
is MS tools are not supported).


> 2) It seems that scons does not interfare with numpy.distutils much.
> If this is true and numpy/scipy builds will not break when scons is
> not installed then I think you could continue the scons support
> development in trunk.
It won't break if scons is not installed because scons sources are 
copied into the branch. Scons developers explicitely support this:

http://www.scons.org/faq.php#SS_3_3

(AFAIK, it does not pose any problem license-wise, since scons is new 
BSD license; it adds ~350 kb of compressed source code to numpy).
>
> 3) In future, if we are going to replace using distutils with scons
> then all numpy/scipy need SConstruct scripts. I think one can implement
> these scripts already now and use, say, setupscons.py, containing only
>
> def configuration(parent_package='',top_path=None):
>      from numpy.distutils.misc_util import Configuration
>      config = Configuration('packagename',parent_package,top_path)
>      config.add_sconscript('SConstruct')
>      return config
> if __name__ == '__main__':
>      from numpy.distutils.core import setup
>      setup(configuration=configuration)
>
> to build the packages. Or can scons already be used with only
> SConstruct script and setupscons.py are not needed? Implementing these 
> scripts now would give a good idea what features are required in using
> scons to build numpy/scipy packages.
> Also, it will prove that scons can replace numpy.distutils in future,
That's what I had in mind (setupscons.py). I don't see any big reasons 
for not merging the current branch (since it is optional and should not 
break anything).

Now, concerning migrating all compiled extensions to scons, I would 
prefer avoiding doing it in the trunk; but I heard horror stories about 
subversion and merging, so maybe staying outside the trunk is too risky 
? Also, when I said that I don't see big problems to replace distutils 
for compiled extensions, I lied by over-simplification. If scons is used 
for compiled extension, with the current design, distutils will call 
scons for each package. Calling scons is expensive (importing many 
modules, etc...: this easily takes around one second for each non 
trivial Sconscript), and also, because each of them is independent, we 
may check several times for the same thing (fortran compiler, etc...), 
which would really add up to the build time.

I see two approaches here:
    - do not care about it because numpy is unlikely to become really bigger
    - considering that numpy is really one big package (contrary to 
scipy which, at least in my understanding, is gearing toward less 
inter-package dependencies ?), we should only have one big sconscript 
for configuration (checking blas/lapack, compilers, etc...), and use 
scons recursively. In this case, it should not be much slower than the 
current system.
>
> 4) though, we cannot remove numpy.distutils for backward compatibility
> with software using numpy.distutils. However, it should be possible
> to enhance Configuration class to generate the corresponding SConstruct
> scripts. It will save some work when converting configuration() 
> functions to SConstruct scripts.
Could you elaborate on this point ? I am not sure what you mean by 
generating Configuration class ?

cheers,

David


More information about the Numpy-discussion mailing list