[AstroPy] Co-ordinating Python astronomy libraries?

Thomas Robitaille thomas.robitaille@gmail....
Tue Jul 6 22:15:30 CDT 2010


> What I think we want from the users' perspective is an ability to say,
> *in the terms of their system's native package manager*, any of:
> 
> give me this/these specific package(s)
> give me this/these group(s) of packages
> give me all the packages
> 
> Then, we can publish metapackages that depend on groups of add-ons by
> topic (file format readers, coordinates/WCS/time, measurement
> extraction, spectral modeling, orbits, planets, galaxies, stars, etc.)
> and one that depends on all of the group packages, and viola, you can
> have Sage, EPD, or whatever fall out as a trivial consequence.

This is a very good idea - it is similar to what I was thinking of when I mentioned a hierarchical setup, but I went too far because - as you pointed out - a full Python distribution would be a nightmare to maintain. What you suggest is something more manageable, and in the end encourages better practices (and after all, who needs yet another full python distribution on their machine, besides e.g. the system python, scisoft, macports, etc.!)

Is there a standard way of combining separate packages, potentially developed in different repositories, into a single meta-package for distribution to the user? Does distutils or distribute allow something like this by default?

Cheers,

Thomas

> What is critical here is something you did not mention in your
> posting, and that is solving the current difficulty of building a good
> Python stack from OS packages (e.g., debs), or even from package
> sources.  The problem is that some of the packages do not play well
> with others.  For example, HDF5 libraries were particularly
> problematic a year ago, and often compiled code complained because two
> different packages wanted different, specific versions of the same C
> or FORTRAN add-on library.  Some of the code plain didn't work as
> advertized, or at all.  In the end, it takes my very skilled system
> manager more than a week to do it, each time we do it, which is about
> once a year.
> 
> The root of the problem is that there was no centralized build and
> test suite, nobody managing unified, integrated build testing and
> resolving the problems with the code maintainers.
> 
> A year ago at SciPy'09, I pointed people to the build and test suite
> NSF requires for all their software projects.  I think a few people
> looked at it at the time.  Configured correctly (which takes work), it
> will build packages on umpteen different linuxes and a few other
> systems and package them in the native format.
> 
> I think that the extended SciPy stack as a whole should be organized
> around such a system, but there seems to be little taste for it among
> the developer-heavy SciPy leadership.  I think we can have a bit more
> practical vision, and at least for our own stuff (loosely defined) we
> should organize around principles that include these:
> 
> - build and test everything frequently
> - manage the namespace so that all can play together
>  - we really need to get everyone to agree to this or there will be
>    conflicts, it's only a matter of time
> - produce LPUs (least packageable units) in binary format for all OSes
> - produce by topical meta-packages
> - produce one or more mega-packages that are equivalent to Sage or EPD
> - do it all for the native installers of all OSes
> - provide and enforce standards for docs and licenses
> - review the code
> - plan together and put out RFPs for needed codes
> - make sure procedures don't stifle innovators
> - document the whole, but lightly
> - provide a web community site for discussions, examples, reviews of code
> - locate and manage it so that it is owned by the community and
>  survives long-term
>  - ensure all jobs doable by at least 3 people
>  - document procedures well
>  - have formalized community governance and leadership
>  - have a solid funding model
> - agree to hang together, even if you don't like something!
> 
> With those (and perhaps other) goals in mind, we should then look at
> decisions like where/how to host it and what kind of wiki to use.
> 
> Also, I keep thinking that this is best solved by joining forces with
> other scientific communities.  The build-and-test part is hard, but
> once implemented, it scales fantastically.  Again, all of SciPy should
> be doing this.  We should at least build so that if others want to
> join, they have a place to fit in.  This will need to be considered
> when doing package naming conventions.  Generic names should be
> avoided so that two can play in the same sandbox.
> 
> Even if we don't do the full thing from the start, we should plan it
> out and build as though that's where we're eventually going.
> 
> AAS splinter meeting, anyone?
> 
> --jh--
> _______________________________________________
> AstroPy mailing list
> AstroPy@scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/astropy/attachments/20100706/2f281fd3/attachment.html 


More information about the AstroPy mailing list