[SciPy-dev] Technology Previews and Sphinx Docstring Annotations

Jarrod Millman millman@berkeley....
Tue Nov 4 11:25:20 CST 2008

I absolutely agree with the ideas presented about scikits and look
forward to seeing the numerous scikits improvements.  I feel that I
have gotten into a discussion where the counter argument to what I am
proposing is something I strongly support.  I also feel that the
counterargument doesn't directly address my concern; but it may be
that I am simply perceiving a problem that no one else believes

I am heading out for a week long vacation, so this will be my last
email on this.  So let me briefly state my concern:

 1. I think it would be great to use the scikits as a staging ground
for code that is going to be introduced into scipy.  However, I don't
think that this has ever happened.  If some one can come up with an
instance, I am happy to be found wrong.  Even if it does become a
staging ground for scipy, I would be uncomfortable requiring that
package developer's who have written mature code that they wish to
contribute to scipy (e.g., Damian's clustering code, PyDSTools, etc.)
first make their code a scikit.  I look forward to all the great
improvements to the scikits infrastructure. However, I would like to
see people stay focused on scipy for just a little bit longer.  We
haven't had a release in quite some time and there is a lot of work
left to be done.  In fact most of the suggestions that have been made
for improving the scikits infrastructure would be great additions to
the numpy/scipy infrastructure.  I look forward to seeing these
improvements, so please don't feel that I am arguing against them.

2. The problem I am raising is how do we best bring in new code to
scipy regardless of whether it comes from scikits or some other
source.  Let's imagine for the sake of argument that the best
imaginable infrastructure for scikits exists.  Will it entirely remove
the need for us to look at the new code's APIs?  Will it make it
easier for all the scipy developer's to test out the scikits?  I am
pretty confident that I can install the scikits (in fact I doubt that
I would even use the scikits installers), but I don't.  I have enough
work just tracking development of numpy, scipy, and the other projects
that I follow.

3. We have limited developer's resources.  This is the main reason
that I think new code isn't better reviewed.  Having more projects to
follow and more infrastructure to develop isn't going to necessarily
solve that issue.

4. The proposal I am making directly addresses code that has currently
been added to the scipy trunk.  Rather than releasing 0.7 and being
relatively stuck with the design decisions and API choices of the
authors, I am proposing a middle ground.  I am not interested in
scipy.preview becoming a dumping ground in the future.  And I believe
we could devise policies to ensure that it doesn't.

Does everyone feel confident that all the APIs for packages included
in the trunk since the last scipy release are stable and won't need to
change?  If you think that they might need to change what do you
propose to do?  We could pull all the new packages and make them
scikits.  We could delay the release and review them all.  We could
release the code and just live with it.  But are those the only
choices we have?  Could there be something like scipy.preview?

OK, so here is my concern one last time before I take off for
vacation.  Since the 0.6 release we have decided to include a number
of new packages, modules, etc (e.g., radial basis functions, sparse
matrices, hierarchical clustering).  My feeling is that we all agreed
that we should include this code and we are all happy to have done it.
 I don't feel that there has been a thorough code review of any of the
code by the scipy developer community.

Now we are preparing to release 0.7.  Once that happens are we
prepared to effectively freeze the API for all that new code?  Do we
intend to accept that we will need two minor releases to implement any
API changes?

Is there a concern that the development cycle for scipy is too slow?
That is my feeling.  I would like to see scipy development become much
more rapid now.  I think that my modest proposal would help in that
direction, but we will need to do much more.  Automated nightly builds
would be a great step.  Better web infrastructure to attract new
developers would help.  But what I am proposing would require little
effort and could basically be set up in a day or so if we agree that
it would be useful.

I called my proposal "modest" because I don't intend it to radically
change how we develop scipy.  I was imagining that we would be able to
work out fairly rigorous criteria for inclusion in scipy.preview
(please see my proposal if you haven't looked at it yet:
 I am open to requiring that code first become a scikit before moving
to scipy.preview.

At this point, it sounds like there is very little interest in
pursuing an idea like scipy.preview.  So unless I find that when I
return in a week that there has been more interest I will drop it.
Hopefully, the discussion to improve scikits will continue.  I look
forward to seeing it improve.  If anyone is interested in editing the
technology preview proposal, please feel free it is checked into the
scipy trunk.  However, I ask that if you wish to propose improvements
to scikits that you start a second sep.


Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014

More information about the Scipy-dev mailing list