[Numpy-discussion] Numpy governance update

[Matthew Brett]

Hi Ben,

On Thu, Feb 16, 2012 at 9:54 PM, Benjamin Root <ben.root@ou.edu> wrote:
>
>
> On Thursday, February 16, 2012, John Hunter wrote:
>>
>>
>>
>> On Thu, Feb 16, 2012 at 7:26 PM, Alan G Isaac <alan.isaac@gmail.com>
>> wrote:
>>>
>>> On 2/16/2012 7:22 PM, Matthew Brett wrote:
>>> > This has not been an encouraging episode in striving for consensus.
>>>
>>> I disagree.
>>> Failure to reach consensus does not imply lack of striving.
>>>
>>
>> Hey Alan, thanks for your thoughtful and nuanced views.  I agree  with
>> everything you've said, but have a few additional points.
>>
>> At the risk of wading into a thread that has grown far too long, and
>> echoing Eric's comments that the idea of governance is murky at best
>> when there is no provision for enforceability, I have a few comments.
>> Full disclosure: Travis has asked me and I have agreed to to serve on
>> a board for "numfocus", the not-for-profit arm of his efforts to
>> promote numpy and related tools.  Although I have no special numpy
>> developer chops, as the original author of matplotlib, which is one of
>> the leading "numpy clients", he asked me to join his organization as a
>> "community representative".  I support his efforts, and so agreed to
>> join the numfocus board.
>>
>> My first and most important point is that the subtext of many postings
>> here
>> about the fear of undue and inappropriate influence of Continuum under
>> Travis' leadership is far overblown.  Travis created numpy -- it is
>> his baby.  Undeniably, he created it by standing on the shoulders of
>> giants: Jim Hugunin, Paul Dubois, Perry Greenfield and his team, and
>> many others.  But the idea that we need to guard against the
>> possibility that his corporate interests will compromise his interests
>> in "what is best for numpy" is academic at best.
>>
>> As someone who has created a significant project in the realm of
>> "scientific computing in Python", I can tell you that it is something
>> I take quite a bit of pride in and it is very important to me that the
>> project thrives as it was intended to: as a free, open-source,
>> best-practice way of doing science.  I know Travis well enough to know
>> he feels the same way -- numpy doing well is *at least* important to
>> him his company doing well.  All of his recent actions to start a
>> company and foundation which focuses resources on numpy and related
>> tools reinforce that view.  If he had a different temperament, he
>> wouldn't have devoted five to ten years of is life to Numeric, scipy
>> and numpy.  He is a BDFL for a reason: he has earned our trust.
>>
>> And he has proven his ability to lead when *almost everyone* was
>> against him.  At the height of the Numeric/numarray split, and I was
>> deeply involved in this as the mpl author because we had a "numerix"
>> compatibility layer to allow users to use one or the other, Travis
>> proposed writing numpy to solve both camp's problems.  I really can't
>> remember a single individual who supported him.  What I remember is
>> the cacophony of voices who though this was a bad idea, because of the
>> "third fork" problem.  But Travis forged ahead, on his own, wrote
>> numpy, and re-united the Numeric and numarray camps.  And
>> all-the-while he maintained his friendship with the numarray
>> developers (Perry Greenfield who led the numarray development effort
>> has also been invited by Travis to the numfocus board, as has Fernando
>> Perez and Jarrod Millman).  Although MPL at the time agreed to support
>> a third version in its numerix compatibility layer for numpy, I can
>> thankfully say we have since dropped support for the compatibility
>> layer entirely as we all use numpy now.  This to me is the distilled
>> essence of leadership, against the voices of the masses, and it bears
>> remembering.
>>
>> I have two more points I want to make: one is on democracy, and one is
>> on corporate control.  On corporate control: there have been a number
>> of posts in this thread about the worries and dangers that Continuum
>> poses as the corporate sponser of numpy development, about how this
>> may cause numpy to shift from a model of a few loosely connected,
>> decentralized cadre of volunteers to a centrally controlled steering
>> committee of programmers who are controlled by corporate headquarters
>> and who make all their decisions around the water cooler unobserved by
>> the community of users.
>>
>> I want to make a connection to something that happened in the history
>> of matplotlib development, something that is not strictly analogous
>> but I think close enough to be informative.  Sometime around 2005,
>> Perry Greenfield, who heads the development team of the Space
>> Telescope Science Institute (STScI) that is charged with processing
>> the Hubble image pipeline, emailed me that he was considering using
>> matplotlib as their primary image visualization tool.  I can't tell
>> you how excited I was at the time.  The idea of having institutional
>> sponsorship from someone as prestigious and resourceful as STScI was
>> hugely motivating.  I worked feverishly for months to add stuff they
>> needed: better rendering, better image support, mathtext and lots
>> more.  But more importantly, Perry was offering to bring institutional
>> support to my project: well qualified full-time employees who
>> dedicated a significant part of their time to matplotlib
>> development. He had done this before with numarray development, and
>> the contributions of his team are enormous.  Many mpl features owe
>> their support to institutional sopnsership: Perry's group deserves the
>> biggest props, but Ted Drain's group at the JPL and corporate sponsors
>> as well are on the list.
>>
>> What I want you to think about are the parallels between Perry and his
>> team joining matplotlib's development effort and Continuum's stated
>> desire to contribute to numpy development.  Yes, STScI is a
>> not-for-profit entity operated by NASA, and Continuum is a
>> for-profit-entity with a not-for-profit arm (numfocus).  But the
>> differences are not so great in my experience.  Both for-profits and
>> not-for-profits experience institutional pressures to get code out on
>> a deadline.  In fact, perhaps our "finest hour" in matplotlib
>> development came as a result of one of out not-for-profit client's
>> deadlines.  My favorite story is when the Jet Propulsion Labs at
>> Caltech emailed me about the inadequacy of our ellipse approximations,
>> and gave us the constraint that the Mars Rover was scheduled to land
>> in the next few months.  Talk about a hard deadline!  Michael
>> Droettboom, under Perry's direction, implemented a
>> "8-cubic-spline-approximation-to-curves-in-the-viewport" solution that
>> I honestly think gives matplotlib the *best* approximation to such
>> curves anywhere.  Period.  Institutional deadlines to get working code
>> into the codebase, whether from a for-profit or not-for-profit entity,
>> and usually are a good thing. It may not be perfect going in, but it is
>> usually better for being there.
>>
>> That is one example from matplotlib's history that illustrates the
>> benefit of institutional sponsers in a project.  In this example, the
>> organizational goal -- getting the Rover to land without crashing -- is
>> one we can all relate to and support.  And the resolution to the story,
>> in which a heroically talented developer (Michael D) steps up to
>> solve the problem, is one we can all aspire to.  But the essential
>> ingredients of the story are not so different from what we face here:
>> an organization needs to solve a problem on a deadline; another
>> organization, possibly related, has the resources to get the job done;
>> all efforts are contributed to the public domain.
>>
>> Now that brings me to my final and perhaps most controverisal point.
>> I don't believe democracy is the right solution for most open source
>> problems.  As exhibit A, I reference the birth of numpy itself that I
>> discussed above.  Numpy would have never happened if community input
>> were considered.  I'm pretty sure that all of us that were there at
>> the time can attest to this.
>>
>> Democracy is something that many of us have grown up by default to
>> consider as the right solution to many, if not most or, problems of
>> governance.  I believe it is a solution to a specific problem of
>> governance.  I do not believe democracy is a panacea or an ideal
>> solution for most problems: rather it is the right solution for which
>> the consequences of failure are too high.  In a state (by which I mean
>> a government with a power to subject its people to its will by force
>> of arms) where the consequences of failure to submit include the
>> death, dismemberment, or imprisonment of dissenters, democracy is a
>> safeguard against the excesses of the powerful.  Generally, there is
>> no reason to believe that the simple majority of people polled is the
>> "best" or "right" answer, but there is also no reason to believe that
>> those who hold power will rule beneficiently.  The democratic ability
>> of the people to check to the rule of the few and powerful is
>> essential to insure the survival of the minority.
>>
>> In open source software development, we face none of these problems.
>> Our power to fork is precisely the power the minority in a tyranical
>> democracy lacks: noone will kill us for going off the reservation.  We
>> are free to use the product or not, to modify it or not, to enhance it
>> or not.
>>
>> The power to fork is not abstract: it is essential.  matplotlib, and
>> chaco, both rely *heavily* on agg, the Antigrain C++ rendering
>> library.  At some point many years ago, Maxim, the author of Agg,
>> decided to change the license of Agg (circa version 2.5) to GPL rather
>> than BSD.  Obviously, this was a non-starter for projects like mpl,
>> scipy and chaco which assumed BSD licensing terms.  Unfortunately,
>> Maxim had a new employer which appeared to us to be dictating the
>> terms and our best arguments fell on deaf ears.  No matter: mpl and
>> Enthought chaco have continued to ship agg 2.4, pre-GPL, and I think
>> that less than 1% of our users have even noticed.  Yes, we forked the
>> project, and yes, noone has noticed.  To me this is the ultimate
>> reason why governance of open source, free projects does not need to
>> be democratic.  As painful as a fork may be, it is the ultimate
>> antidote to a leader who may not have your interests in mind.  It is
>> an antidote that we citizens in a state government may not have.
>>
>> It is true that numpy exists in a privileged position in a way that
>> matplotlib or scipy does not.  Numpy is the core.  Yes, Continuum is
>> different than STScI because Travis is both the lead of Numpy and the
>> lead of the company sponsoring numpy.  These are important
>> differences.  In the worst cases, we might imagine that these
>> differences will negatively impact numpy and associated tools.  But
>> these worst case scenarios that we imagine will most likely simply
>> distract us from what is going on: Travis, one of the most prolific
>> and valuable contributers to the scientific python community, has
>> decided to refocus his efforts to do more.  And that is a very happy
>> moment for all of us.
>
>
>
> John,
>
> Thank you for taking the time to share your perspective.  As always, it is
> very interesting.  I have only been involved with numpy for the past 3
> years. From my perspective, Chuck has been BDFL, in a sense.  It is hard for
> me to imagine python without numpy, and it is difficult to fully appreciate
> the effort Travis has put in. Perspectives like your are useful for this.
>
> So, power is derived by a mandate from the masses. That part is always
> democratic. However, the form of governance is not required to be
> democratic.
>
> If we are agreeing to Travis being BDF12, then that is just as valid in my
> mind as us agreeing to some other structure. What is important is the
> community coming together and agreeing on this.  Would that be fine for
> everybody?

Actually this is more to thank you for the tone of your email than
anything else.   I know we've disagreed from time to time, but I do
appreciate your constant and kind attempts to bring people together.

I think there's no controversy about Travis being BDF$N$ and yes, I
agree, to specify that is a step forward.

Best,

Matthew