[Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
perry at stsci.edu
Wed Feb 9 08:49:36 CST 2005
I thought I would clarify some historical issues (at least as far as
I recall them) in helping understand how things got the way they are
Travis Oliphant wrote:
> Francesc Altet wrote:
> I think numarray has made some incrdedible strides in showing what the
> numeric object needs to be and in implementing some really neat
> functionality. I just think its combination of Python and C code must
> be redone to overcome the speed issues that have arisen. My opinion
> after perusing the numarray code is that it would be easier (for me
> anyway) to adjust Numeric to support the features of numarray, than to
> re-write and re-organize the relevant sections of numarray code. One of
> the advantages of Numeric is it's tight implementation that added only
> two fundamental types, both written entirely in C. I was hoping that
> the Python dependencies for the fundamental types would fade as numarray
> matured, but it appears to me that this is not going to happen.
When we started this, one of those that suggested putting much of
the code in Python was Guido himself if I recall correctly (I'd have
to dig up the message he wrote on the subject to see what exactly he
said). That was one of the factors influencing us to go that route
(as well as other mentioned below).
> I did not have the time in the past to deal with this. I wish I had
> looked at it more closely two years ago. If I had done this I would
> have seen how to support the features that Perry wanted without
> completely re-writing everything. But, then again, Python 2.2 changed
> what is possible on the C level and that has had an impact on the
Indeed, numarray was started well before Python 2.2 and was another
it wasn't done in C. Knowing what would have been available in 2.2
would likely have changed the approach used.
> >- Memory-mapped objects: Allow working with on-disk numarray
> objects like if
> > they were in-memory.
> Numeric3 supports this cleanly and old Numeric did too (there was a
> memory-mapped module), it's just that byteswapping, and alignment had to
> be done manually.
Just to clarify (it may not be immediately apparent to those who haven't
had to deal with it) many memory mapped files do not use the machine
representation (as is often the case for astronomical data files).
Memory mapping isn't nearly as useful if one has to create temporaries
to handles these cases.
> >- RecArrays: Objects that allow to deal with heterogeneous datasets
> > (tables) in an efficient manner. This ought to be very
> beneficial in many
> > fields.
> Heterogeneous arrays is the big one for old Numeric. It is a good
> idea. In Numeric3 it has required far fewer changes than I had at first
> Numeric has had this in mind for some time. In fact the early Numeric
> developers were quite instrumental in getting significant changes into
> Python istelf, including Complex Objects, Ellipses, and Extended
> Slicing. Guido was quite keen on the idea of including Numeric at one
> point. Our infighting made him lose interest I think. So claiming
> this as an advantage of numarray over Numeric is simply inaccurate.
Actually, as I understand it, Guido had ruled out including Numeric well
before numarray even got started. I can't claim to know exactly his
reasons (I've heard various things such as he looked at the code and
didn't like it, or Paul Dubois and others advised him against it; I
can't say exactly why), but I am sure that that decision was made
before numarray. No doubt the split has prevented any further
consideration of inclusion.
More information about the Numpy-discussion