[Numpy-discussion] "expected a single-segment buffer object"

Robert Kern robert.kern@gmail....
Wed Jul 9 22:03:18 CDT 2008


On Wed, Jul 9, 2008 at 21:29, Anne Archibald <peridot.faceted@gmail.com> wrote:
> 2008/7/9 Robert Kern <robert.kern@gmail.com>:
>
>> Yes, the buffer interface, at least the subset that ndarray()
>> consumes, requires that all of the data be contiguous in memory.
>> array_as_buffer() checks for that using PyArray_ISONE_SEGMENT(), which
>> looks like this:
>>
>> #define PyArray_ISONESEGMENT(m) (PyArray_NDIM(m) == 0 ||                      \
>>                                 PyArray_CHKFLAGS(m, NPY_CONTIGUOUS) ||       \
>>                                 PyArray_CHKFLAGS(m, NPY_FORTRAN))
>>
>> Trying to get a buffer object from anything that is neither C- or
>> Fortran-contiguous will fail. E.g.
>>
>> In [1]: from numpy import *
>>
>> In [2]: A = arange(10)
>>
>> In [3]: B = A[::2]
>>
>> In [4]: ndarray(strides=B.strides, shape=B.shape, buffer=B, dtype=B.dtype)
>> ---------------------------------------------------------------------------
>> TypeError                                 Traceback (most recent call last)
>>
>> /Users/rkern/today/<ipython console> in <module>()
>>
>> TypeError: expected a single-segment buffer object
>
> Is this really necessary? What does making this restriction gain? It
> certainly means that many arrays whose storage is a contiguous block
> of memory can still not be used (just permute the axes of a 3d array,
> say; it may even be possible for an array to be in C contiguous order
> but for the flag not to be set), but how is one to construct exotic
> slices of an array that is strided in memory? (The real part of a
> complex array, say.)

Because that's just what a buffer= argument *is*. It is not a place
for presenting the starting pointer to exotically-strided memory. Use
__array_interface__s to describe the full range of representable
memory. See below.

> I suppose one could follow the linked list of .bases up to the
> original ndarray, which should normally be C- or Fortran-contiguous,
> then work out the offset, but even this may not always work: what if
> the original array was constructed with non-C-contiguous strides from
> some preexisting buffer?
>
> If the concern is that this allows users to shoot themselves in the
> foot, it's worth noting that even with the current setup you can
> easily fabricate strides and shapes that go outside the allocated part
> of memory.
>
>> What is the use case, here? One rarely has to use the ndarray
>> constructor by itself. For example, the result you seem to want from
>> the call you make above can be done just fine with .view().
>
> I was presenting a simple example. I was actually trying to use
> zero-strided arrays to implement broadcasting.  The code was rather
> long, but essentially what it was meant to do was generate a view of
> an array in which an axis of length one had been replaced by an axis
> of length m with stride zero. (The point of all this was to create a
> class like vectorize that was suitable for use on, for example,
> np.linalg.inv().) But I also ran into this problem while writing
> segmentaxis.py, the code to produce a "matrix" of sliding windows.
> (See http://www.scipy.org/Cookbook/SegmentAxis .) There I caught the
> exception and copied the array (unnecessarily) if this came up.

I was about a week ahead of you. See numpy/lib/stride_tricks.py in the trunk.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco


More information about the Numpy-discussion mailing list