[Numpy-discussion] Build Process and Version Control

Ralf Gommers ralf.gommers@googlemail....
Mon Oct 25 09:13:06 CDT 2010

Hi Friedrich,

On Mon, Oct 25, 2010 at 6:08 PM, Friedrich Romstedt
<friedrichromstedt@gmail.com> wrote:
> Hi,
> here is a summary of what we got so far with the Mac OS X build process:

Thanks for the detailed info on what you are doing. First maybe a
general comment: the current release scripts work quite well, so it's
probably not a very good idea to throw all that out. Also, I've tried
to improve the docs on how the binaries are built and make them as
complete as possible over the last half year, so I would appreciate
you and Vincent pointing out any gaps or creating some patches to
those docs.

> -  We found it useful to put
> /Library/Frameworks/Python.framework/Versions/2.x under version
> control with git.  This lets us get rid of the paver bootstrap thingy,
> since we just switch to a branch via ``git checkout numpy``.
> We can have a full-blown polluted Python and a minimial numpy Python in
> parallel, and we can revert to the built numpy libraries when we find
> some issue later.  We will note the sha of the numpy system commit in
> the commits in the numpy build git repo.

Not sure about the motivation here. Paver and virtualenv (the
"bootstrap thingy") are widely used and relatively mature tools, why
hack something together that does the same job?

> -  We also modified the Pythons via a print statement in
> sitecustomize.py to tell what numpy is running on startup, this goes
> to the logfiles etc., it also tells the branch family of the system
> git repo (hardcoded).

First time I heard about this sitecustomize.py, sounds like not the
most robust thing I've ever heard of...

> -  We established a numpy git "build repo".  It'll contain all the
> dmgs and so on, along with the log files of the build process.

I like the idea of recording all logs and having them accessible for
later reference. Very large binaries under version control, less so.

>  I will push soon.  Its location is http://github.com/friedrichromstedt/numpy.
>  We tried hard to make good design choices about the branches, but
> this is discussable, and might be easy to revert in this early stage
> where noone has fetched so far.
> -  We also set up the "numpy build site", it's here:
> http://vincentdavis.info/, or for the hard-core people here :-)
> http://vincentdavis.info/numpy-build-doc.  It's not that hard-core at
> all.  Just a Sphinx site where we document what we're doing to our
> build system.  Atm I'm not *that* satisfied with it, because I tend to
> document *too much* when new to some subject.  I believe it will turn
> out as a well-readable detailed instruction for ourselves, usable to
> others too, and then we just crossref that instruction instead of
> writing down all the stuff again and again.
> Currently the numpy-build-docs are in another repo,
> http://github.com/friedrichromstedt/numpy-build-doc, I'm not sure if
> it is useful to merge this into my numpy repo.  Pro: Users of the
> numpy repo will get all the stuff with little overhead.  Con: Users of
> the numpy-build-doc repo will have *lots* of overhead.  But I guess
> the latter are rather rare.  Please comment.

Why not add to http://projects.scipy.org/numpy/wiki/MakingReleases? Or
otherwise a new plain reST file under doc/. Between the docs in the
git repo, the Trac wiki and the wiki on scipy.org I think we have
enough places to add docs. And maintain them...

Again, I appreciate your work and am not trying to stop you from
making improvements. But please try to improve what's already there,
or at least first point out why that is difficult so we can discuss.


> The process is open for others to chime in.  The Sphinx site is ready
> to host also build logs of other systems.  The sources are included in
> the git repo (numpy-build-docs), together with a current html build.
> Please use the Sphinx site online
> (http://vincentdavis.info/numpy-build-doc) and not the git repo, but
> I'll push soon.
> Documentation goes via Python functional programming, but this are
> details.  In fact, Sphinx just documents Python functions.  We are
> working towards an automatisation of build process steps using this
> Python functions.  Yet another build system.  But this is future.
> Please refrain from fetching anything until I did the authoritative
> push, so that you'll have a working system, which is at least beta and
> not rather alpha like the current things I put on github.
> Friedrich
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

More information about the NumPy-Discussion mailing list