[Numpy-discussion] Trying out Numeric3
oliphant at ee.byu.edu
Fri Mar 25 11:33:13 CST 2005
Chris Barker wrote:
> Scott Gilbert wrote:
>> I don't know what particular meta data you plan to store with the buffer
>> itself, and I'm going to resist the urge to guess. You probably have
>> very good use cases. What are you planning?
> I don't know what Travis has in mind, but I thought I'd bring up a use
> case that I think provides some of the motivation for this.
This is exactly the kind of thing I mean. One of the reasons for
putting Numeric in the core is so other extension writers can use it
reliably. But, really, a better solution is to create ways to deal with
each others memory reliably. I think the bytes object Scott proposed
is really very close to what is needed. Extension writers will likely
need some additional information in order to use somebody else's byte
object well. What information is needed can vary (but it should be
standard for different types of objects). You can already get at the
memory directly through the buffer protocol. But,
I think pickling could be handled efficiently with a single new opcode
very similar to a string but instead creating a bytes object on
unpickling. Then, an array could use the memory of that bytes object
(instead of creating it's own and copying). This would be very easy to
I really believe we need to push forward the bytes object. This
attitude towards the buffer interface (due to an easily fixed problem)
is really disquieting. I could spend time pushing the bytes object,
but it would take away from the work I'm currently doing. This is
something that we really need some help with.
Scott's PEP is quite good and gives an outline for how to do this, so
except for pickling you don't even need to be an expert to get something
done. You could start with the outline of numarray's memory object
(getting rid of the Int64 stuff in it), and proceeding from there.
It would probably take a week to get it done.
> My understanding of this memory object is that an extension like
> wxPython wouldn't not need to know about Numeric, but could simply get
> the memory Object, and there would be enough meta-data with it to
> typecast and loop through the data. I'm a bit skeptical about how this
> would work. It seems that the metadata required would be the full set
> of stuff in an array Object already:
> This could be made a bit simpler by allowing only contiguous arrays,
> but then there would need to be a contiguous flag.
I'm thinking just contiguous arrays would be passed. While Numeric does
support the multi-segment buffer interface. I doubt extension writers
want to try and understand how to deal with it. I think it would be
too much of a burden to other extensions if the array they saw was not
contiguous. Even internal to Numeric, discontiguous arrays are made
contiguous all the time (although the new iterator in Numeric3 makes it
much easier for a programmer to deal with discontiguous arrays).
> To make use of this, wxPython would have to know a fair bit about
> Numeric Arrays anyway, so that it can check to see if the data is
> appropriate. I guess the advantage is that while the wxPython code
> would have to know about Numeric arrays, it wouldn't have to include
> Numeric headers or code.
It only has to know the shape and type (typechar and itemsize) of the
array if we stick to contiguous arrays (this is where the typechar
becomes very valuable).
I still think the bytes object is a really, really good idea. Python
needs it very badly. If every extension module that allocated memory
went through a bytes object instead, then a lot of unnecessary copying
could be minimized.
More information about the Numpy-discussion