[SciPy-dev] Formal Review Process

Robert Kern kern at ugcs.caltech.edu
Sat Sep 20 04:36:04 CDT 2003


At both SciPy conferences, Eric emphasized the need for a peer-review
process for potential contributions to SciPy. I recalled that the Boost
community established a formal review process for contributions to their
set of libraries, and it seems to work well for them. I got bored of
writing a history paper, so I looked at their process and wrote up a
proposal for SciPy (liberally using parts of the Boost documents where
noted).

Please comment on, add to, delete from, and otherwise contribute to the
following proposal. You can email me personally as well as respond to
the list.

As he announced at SciPy '03, Chad Netzer has a wavelets module he would
like to submit. If we can hammer out some consensus (enough for a test
run at least), then Chad can be our first sacrif^Wclient. I am sure he
is as thrilled to learn of this possibility as I am to raise it.

The text is included below. HTML and text versions can also be found in
the following URLs:

http://www.ugcs.caltech.edu/~kern/scipy-review.html
http://www.ugcs.caltech.edu/~kern/scipy-review.txt


A Proposal for a SciPy Module Submission Review Process
=======================================================

:Author: Robert Kern (parts by the Boost_ group and others__)
:Version: 0.0
:Date: 2003-09-20

The following proposal is inspired by (and indeed partly outright lifted from)
the `Boost Formal Review Process`_ and `Submission Process`_. It is written
using ReStructuredText_.

.. _Boost: http://www.boost.org
__ Acknowledgements_
.. _Boost Formal Review Process: 
    http://www.boost.org/more/formal_review_process.htm
.. _Submission Process: http://www.boost.org/more/submission_process.htm
.. _ReStructuredText : http://docutils.sourceforge.net


Rationale
---------

1. The SciPy package requires a peer-review process to examine new module
   submissions. The expertise of the CVS maintainer is limited, and the
   number of fields for which one could write a SciPy module is large.

2. A formal procedure for the review process makes it easier for newcomers to
   know exactly what they have to do to get their contribution into SciPy.
   Furthermore, a formal procedure makes it easier for Enthought to parcel out
   the work to non-Enthought people and even to transition SciPy to a wholly
   community-driven project if that ever becomes a goal.

3. The Boost project had a similar problem, found a solution that worked for
   them, and wrote it up all pretty, so why not steal it? [XXX - or something to
   this effect]

Organization
------------

Mailing List
~~~~~~~~~~~~

Discussion of libraries under formal review shall take place on
scipy-dev at scipy.org . Actual reviews of a candidate module (see section `Reviews`_) 
during its formal review period should begin their Subject header with 
[REVIEW - <name of module>] for the convenience of the Review Manager.

Reviewers who do not wish to subscribe to scipy-dev at scipy.org may use the GMane_
WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not
allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows
posting for this list, either.]

.. _GMane: http://www.gmane.org
.. _WWW: http://news.gmane.org/gmane.comp.python.scientific.devel
.. _Usenet: nntp://news.gmane.org/gmane.comp.python.scientific.devel

Review Wizard
~~~~~~~~~~~~~

(The Review Wizard/Manager sections are only slightly changed from `Boost Formal
Review Process`_.)

The Review Wizard:

  * Maintains a list of review manager volunteers.

  * When a formal review is requested for a module:

    - Assigns a review manager and suggests a schedule.
    
    - Finalizes the schedule, once the review manager verifies the module is
      actually ready for review.

    - Resolves schedule slips or other issues with review managers and submitters.

  * Maintains a schedule of both past and pending reviews.
  
  * Resolves questions from review managers and module submitters, who
    sometimes want a third opinion on questions such as "Should we extend the
    review period because ...?"

  * Monitors the general review process, and makes minor adjustments as needed,
    or queries the list about possible major adjustments.

.. _Boost Formal Review Process: 
   http://www.boost.org/more/formal_review_process.htm

Review Manager
~~~~~~~~~~~~~~

The Review Manager:

  * Checks the submission to make sure it really is complete enough to warrant
    formal review. If necessary, work with the submitter to verify the code
    compiles and runs correctly on several platforms.

  * Finalizes the schedule with the Review Wizard and the submitter.

  * Posts a notice of the review schedule on the appropriate mailing list(s). 

    - The notice should include a brief description of the module and what it
      does, to let readers know if the module is one they are interested in
      reviewing.
 
    - If the module is known to fail on certain platforms, please mention
      them in the review notice so reviewers on those platforms won't waste
      time diagnosing known problems (or alternatively, will test and try to
      resolve the failures).

  * Inspects the SciPy package for modules which may interact with the new
    submission. These potential interactions should be pointed out in the review
    announcement, and the author(s) of these modules should be privately
    notified and urged to participate in the review.

  * Urges people to do reviews if they aren't forthcoming.

  * Follows review discussions regarding the modules, moderating or answering
    questions as needed.

  * Decides if there is consensus to accept the modules, and if there are any
    conditions attached.

  * Posts a message to the appropriate mailing lists informing people of the
    review results.

In other words, it is the Review Manager's responsibility to make sure the
review process works smoothly.

The Process for a Module Submitter
----------------------------------

Determine Interest
~~~~~~~~~~~~~~~~~~

Post a message to scipy-dev at scipy.org to see if there is interest in your
module. Describe what your module does and perhaps include a small snippet
of code that shows how one uses it.

Preliminary Submission
~~~~~~~~~~~~~~~~~~~~~~

Put your (possibly unfinished) module on a website and post its location to
scipy-dev at scipy.org . Contact the maintainers of www.scipy.org for a Zope
account if you need web space for your module.

Submissions should follow the Code Submission Guidelines [XXX - this should
point to something useful. For now, look at the `Code Formatting Guidelines`_
and the `Testing Guidelines`_.]

.. _Code Formatting Guidelines: 
    http://www.scipy.org/site_content/tutorials/formatting_guidelines
.. _Testing Guidelines: 
    http://www.scipy.org/site_content/tutorials/testing_guidelines

Refinement
~~~~~~~~~~

Discuss the module in scipy-dev at scipy.org . Fix bugs; add functionality;
change the interface; write docs/unit tests; do whatever comes out of the
discussion as necessary or beneficial to including the module in SciPy.
Update the preliminary submission until it is ready for formal submission.

Submission for Review
~~~~~~~~~~~~~~~~~~~~~

Package your module submission in a ZIP file or gzipped tar file such that
it can be unpacked in the scipy CVS directory [XXX - what if you need to
patch the setup.py file or scipy_distutils files? Diff file?].

Contact the Review Wizard. He will assign a Review Manager and create a
www.scipy.org Zope account for you. Place your submission in [XXX] on the
www.scipy.org website.

The Review Manager will kick off the reviews on scipy-dev at scipy.org .

Reviews
-------

(Again, taken from `Boost Formal Review Process`_).

Reviews may be submitted to either the list as a whole or the Review Manager
privately.

Your comments may be brief or lengthy, but basically the Review Manager needs
your evaluation of the module. If you identify problems along the way,
please note if they are minor, serious, or showstoppers.

Here are some questions you might want to answer in your review:

* What is your evaluation of the design?

* What is your evaluation of the implementation?

* What is your evaluation of the documentation?

* What is your evaluation of the unit-test coverage?

* What is your evaluation of the potential usefulness of the module?

* Did you try to use the module? On what platform? Did you have any problems?

* How much effort did you put into your evaluation? A glance? A quick reading?
  In-depth study?

* Are you knowledgeable about the problem domain? Are you knowledgeable about
  the programming techniques used? [E.g. perhaps you know nothing about what the
  underlying FORTRAN code does, but you are an expert in wrapping code with
  f2py. Both areas of knowledge are useful for a review.]

And finally, every review should answer this question:

* Do you think the module should be accepted into the SciPy package?  Be sure
  to say this explicitly so that your other comments don't obscure your
  overall opinion.

The module should be stable during the review period (making exception for
serious bug fixes). Modifications suggested during review and accepted by the
submitter should be noted on the mailing list so reviewers can make an
informed decision.

At the end of the review period, the Review Manager will make the final
decision to accept or reject the module for inclusion into SciPy. He will post
this result to the mailing list along with a rationale and any suggestions or
conditions that must be met before the module can be included (or resubmitted
for review, as the case may be).

Acknowledgements
----------------

I would like to thank the Boost community for their well thought-out review
process and the words describing said process. [XXX - Eric, please check if this
is OK with them]

Eric Jones, [XXX - Your Names Here] provided helpful comments and
contributions to this document.

[XXX - Kern's Notes]
--------------------

The Review Wizard/Manager roles will probably be taken by Enthought people or
their designees at least at the start, mostly because they would have to do
similar tasks without a formal review process, so who else would volunteer?

Note the XXX's. They need content or answers.

Boost adds the libraries up for review into a CVS sandbox. We may want to do the
same.

The process as a whole feels a bit heavyweight to me, but there are some
benefits that flow from that. Some nips and tucks couldn't hurt, though.

-- 
Robert Kern
kern at caltech.edu

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter




More information about the Scipy-dev mailing list