[Numpy-discussion] Meta: too many numerical libraries doing the same thing?

Christos Siopis <siopis@umich.edu> siopis at umich.edu
Fri Nov 23 20:59:01 CST 2001

[ This message got longer than i had initially thought, but these thoughts 
  have been bugging me for so long that i cannot resist the temptation to 
  push the send button! Apologies in advance to those not interested...

On Mon, 26 Nov 2001, Martin Wiechert wrote:

> Hi!
> Just an uneducated question.
> Are there any plans to wrap GSL for Numpy2?
> I did not actually try it (It's not Python ;-)),
> but it looks clean and powerful.
> Regards,
> Martin.

I actually think that this question has come up before in this list,
perhaps more than once. And i think it brings up a bigger issue, which is:  
to what extent is it useful for the numerical community to have multiple
numerical libraries, and to what extent does this constitute a waste of

Numpy (Python), PDL (Perl), GSL (C), and a rather large number of other
libraries usually have to re-implement the same old numerical algorithms,
but offered under a different interface each time. However, there is such
a big body of numerical algorithms out there that it's a daunting effort
to integrate them into every language's numerical library (anyone want to
implement LAPACK's functionality in Numpy?) The compromise that is usually
made is to wrap one library around another. While this may be "better than
nothing", it is usually not a pleasant situation as it leads to
inconsistencies in the interface, inconsistencies in the error handling,
difficulties in the installation, problems with licensing,...

Since i have been a beneficiary rather than a contributor to the numerical
open-source community, i feel somewhat hesitant to file this "complaint",
but i really do think that there are relatively few people out there who
are both willing and capable of building quality open-source numerical
software, while there are too many algorithms to implement, so the
community should be vigilant to minimize waste of resources!

Don't take me wrong, i am not saying that Numpy, PDL, GSL & co. should be
somehow "merged" --obviously, one needs different wrappers to call
numerical routines from Python, Perl, C, C++ or Java. But there should be
a way so that the actual *implementation* of the numerical algorithms is
only done once and for all.

So what i envision, in some sense, is a super-library of "all"/as many as
possible numerical algorithms, which will present appropriate (but
consistent) APIs for different programming languages, so that no matter
what language i use, i can expect consistent interface, consistent
numerical behavior, consistent error handling etc. Furthermore, different
levels of access should allow the application developer to access
low-level or high-level routines as needed (and could object orientation
be efficiently built as a higher-level wrapper?)

This way, the programmer won't have to worry whether the secant root
finder that s/he is using handles exceptions well or how NaNs are treated.
Perhaps most importantly, people would feel compelled to go into the pain
of "translating" existing libraries such as LAPACK into this common
framework, because they will know that this will benefit the entire
community and won't go away when the next scripting language du jour
eclipses their current favorite. Over time, this may lead to a truly
precious resource for the numerical community.

Now, i do realize that this may sound like a "holy grail" of numerical
computing, that it is something which is very difficult, if not impossible
to accomplish. It certainly does not seem like a project that the next
ambitious programmer or lab group would want to embark into on a rainy
day. Rather, it would require a number of important requirements and
architectural decisions to be made first, and trade-offs considered. This
would perhaps be best coordinated by the numerical community at large,
perhaps under the auspices of some organization. But this would be time
well-spent, for it would form the foundations on which a truly universal
numerical library could be built. Experience gained from all the numerical
projects to this day would obviously be invaluable in such an endeavor.

I suspect that this list may not be the best place to discuss such a
topic, but i think that some of the most active people in the field lurk
here, and i would love to hear their thoughts and understand why i am
wrong :) If there is a more appropriate forum to discuss such issues, i
would be glad to be pointed to it --in which case, please disregard this

/  Christos Siopis              | Tel    : 734-764-3440       \
/  Postdoctoral Research Fellow |                             \
/  Department of Astronomy      | FAX    : 734-763-6317       \
/  University of Michigan       |                             \
/  Ann Arbor, MI 48109-1090     | E-mail : siopis at umich.edu   \
/  U.S.A.  _____________________|                             \
/         / http://www.astro.lsa.umich.edu/People/siopis.html \

More information about the Numpy-discussion mailing list