[Numpy-discussion] Re: Future directions for SciPy in light of meeting at Berkeley
jh at oobleck.astro.cornell.edu
Wed Mar 9 07:43:13 CST 2005
These were exactly the issues we addressed at SciPy04, and which led
to the ASP project. All of the issues brought up in the current
discussion have already been discussed there, and with largely the
same conclusions. The basic gist is this:
THERE ARE THOUSANDS OF PEOPLE WAITING FOR SCIPY TO REACH CRITICAL MASS!
SciPy will reach the open-source jumping-off point when an outsider
has the following experience: They google, find us, visit us, learn
what they'll be getting, install it trivially, and read a tutorial
that in less than 15 minutes has them plotting their own data. In
that process, which will take less than 45 minutes total, they must
also gain confidence in the solidity and longevity of the software and
find a supportive community. We don't meet all the elements of this
test now. Once we do, people will be ready to jump on and work the
The goal of ASP (Accessible SciPy) is to meet that test. Some of what
we need is being done already, but by a very small number of people.
We need everyone's help to reach a meaningful rate of progress. The
main points and their status:
1. Resolve the numeric/numarray split and get at least stubs for the
basic routines in the Python core. Nothing scares new users more
than instability and uncertainty. Travis O. is now attempting to
incorporate numarray's added features (including much of the code
that implements them) into numeric, and has made a lot of headway.
Perry G. has said that he would switch back to numeric if it did
the things numarray does. I think we can forsee a resolution to
this split in the calendar year IF that effort stays the course.
2. Package it so that it's straightforward to install on all the
popular architectures. Joe Cooper has done a lot here, as have
others. The basic stuff installs trivially on Red Hat versions of
Linux, Windows, and several others (including Debian, I think, and
Mac, modulo the inherent problems people report with the Mac
package managers, which we can do nothing about). Optimized
installs are also available and not all that difficult,
particularly if you're willing to issue a one-line command to
rebuild a source package. For Linux, it was decided to stick with
a core and add-on packages, and to offer umbrella packages that
install common groups of packages through the dependency mechanism
(e.g., for astronomy or biology). The main issue here is not the
packaging, but the documentation, which is trivial to write at this
point. I was able to do a "yum install scipy" at SciPy04, once I
knew where the repository was. It's:
We need someone to write installation notes for each package
manager. We also need umbrella packages.
3. Document it thoroughly for both new and experienced users. Right
now what we have doesn't scratch the surface. I mean no offense to
those who have written what we do have. We need to update that and
to write a lot more and a lot else. Janet Swisher and several
others are ready to dig into this, but we're waiting for the
numeric/numarray split to resolve. A list of needed documents is
in the ASP proposal.
4. Focus new users on a single selection of packages. The variety of
packages available to do a particular task is both a strength and a
weakness. While experienced people will want choice, new users
need simplicity. We will select a single package each application
(like plotting), and will mainly describe those in the
tutorial-level docs. We will not be afraid to change the selection
of packages. You're only a new user once, so it will not affect
you if we switch the docs after you've become experienced. For
example, Matplotlib was selected at the SciPy04 BoF, but if Chaco
ever reaches that level of new-user friendliness, we might switch.
Both packages will of course always be available. Neither needs to
be in the core on Linux and other systems that have package
management. New users will be steered to the "starter" umbrella
package, which will pull in any components that are not in the
core. Enthon will continue to include all the packages in the
world, I'm sure!
5. Provide a web site that is easy to use and that communicates to
each client audience. We (me, Perry, Janet, Jon-Eric) were
actually gearing up to solicit proposals for improving the site and
making it the go-to place for all things numerical in python when
Travis started his work on problem #1. This is the next step, but
we're waiting for item 1 to finish so that we don't distract
everyone's attention from its resolution. Many developers are
interested in contributing here, too. If people feel it's time, we
can begin this process. I just don't want to slow Travis and his
helpers one tiny bit!
6. Catalog all the add-ons and external web sites so that scipy.org
becomes the portal for all things numeric in python. This, at
least, is done, thanks to Fernando Perez. See:
I'll add one more issue:
7. Do something so people who use SciPy, numeric, and numarray
remember that these issues are being worked, and where, and how to
contribute. To that end, all I can do is post periodically about
ASP and encourage you to remember it whenever someone wonders why
we haven't hit critical mass yet. Please visit the ASP wiki. Read
the ASP proposal if you haven't, sign up to do something, and do
it! Right now, a paltry 6 people have signed up to help out.
The ASP proposal is linked in the first paragraph of the wiki.
After giving it some thought, we decided to use scipy-dev at scipy.net as
our mailing list, to avoid cross-posted discussions on the 4 mailing
lists. Please carry on any further discussion there.
More information about the Numpy-discussion