[SciPy-dev] implementing IDL, Matlab, etc. functionality

Joe Harrington jh at oobleck.astro.cornell.edu
Thu May 5 10:51:47 CDT 2005

> In the case of IDL, most algorithms are 
> implemented in single precision floating point.  The Python 
> implementation by default would use double precision, unless we 
> explicitly direct it to do otherwise.  This problem alone can cause much 
> grief, because the IDL version is presumed to be the correct one, until 
> demonstrated otherwise.  (I know this from personal experience.)  So in 
> addition to the language being crappy, do we also want to propagate 
> crappy (i.e. unstable) algorithms too?

Paul, that argument involves both guilt by association and an implicit
assumption that those scientists who *are* guilty will write better
code the second time around.  My experience and what I've been told
directly by dozens of researchers is that they will not write code a
second time, period.  Their main reasons to switch are cost and
freedom, not any unhappiness with IDL.  The balance in their minds
tips to IDL if the cost of rewriting is included.  It tips back to
Python if they have a tool that converts most of their code.  As you
point out, Python defaults to double, so that may improve things a
little for some algorithms.  I doubt it will make a big difference for
most codes, as astronomical data tends to be uncertain no later than
the fourth decimal place, and frequently in the first.

As for the guilt by association, writing in IDL does not make an
algorithm or its implementation bad, nor does writing in Python
automatically make them any better.  Quick-and-dirty, procedural
scientists will write quick-and-dirty, procedural code in Python as
well as in IDL.

On the flip side, I have an algorithm for optimal spectrum extraction
that I can demonstrate is the best available (paper with the
first-ever comparison tests in prep).  It's 4300 lines and took well
over 1000 billable programmer-hours to write.  If I can do an "80%"
translation to Python automatically, spend a few weeks filling in the
external function calls, and run my verification tests to see where
the problems are and fix them, we will have a valuable algorithm that
will attract users to Python.  I don't have the resources to do it by
hand, and I doubt anyone else will do it for me (would STScI?).  In
preparing our paper, we discovered either algorithmic deficiencies
(your "crap") or serious bugs in every other code we tested, including
popular ones like Piskunov's REDUCE and IRTF's SpeXtool.  So, either
we do a converter and you get this and many other tools that will
attract scientists to Python, or we don't, and until Python becomes
the dominant tool (if ever), people will be home-brewing their own
codes for difficult algorithms like optimal spectrum extraction.  And
then you will have real "crap", as you put it, but only necessarily so
in Python.  This is just one of many examples.

It's great that you at STScI are redoing the astron library from the
ground up, but I doubt that you can commit to doing even all of that
work (including astron/contrib).  You certainly can't do everyone's
personal codes, whereas a converter can make a big dent in that.
Hopefully, that will be a big enough dent to induce people to switch.


More information about the Scipy-dev mailing list