[Numpy-discussion] views or copy

[josef.pktd@gmai...]

On Mon, Apr 6, 2009 at 11:34 PM, Robert Kern <robert.kern@gmail.com> wrote:
> On Mon, Apr 6, 2009 at 22:31,  <josef.pktd@gmail.com> wrote:
>> I ran again into a problem where numpy created a view (which I didn't
>> realize) and an operation works differently on the view than if it
>> were a copy.
>>
>> I try to construct an example array, which, however, is only a view
>>
>>>>> x,y = np.mgrid[0:3,0:3]
>>>>> xx = np.vstack((x.flatten(), y.flatten(), np.ones(9))).T
>>>>> xx
>> array([[ 0.,  0.,  1.],
>>       [ 0.,  1.,  1.],
>>       [ 0.,  2.,  1.],
>>       [ 1.,  0.,  1.],
>>       [ 1.,  1.,  1.],
>>       [ 1.,  2.,  1.],
>>       [ 2.,  0.,  1.],
>>       [ 2.,  1.,  1.],
>>       [ 2.,  2.,  1.]])
>>>>> xx.flags
>>  C_CONTIGUOUS : False
>>  F_CONTIGUOUS : True
>>  OWNDATA : False
>>  WRITEABLE : True
>>  ALIGNED : True
>>  UPDATEIFCOPY : False
>>>>> xx.base
>> array([[ 0.,  0.,  0.,  1.,  1.,  1.,  2.,  2.,  2.],
>>       [ 0.,  1.,  2.,  0.,  1.,  2.,  0.,  1.,  2.],
>>       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]])
>>>>> xx == xx.base
>> False
>>
>> When I convert it to a view as a structured array, it produces a
>> strange result. I didn't get what I thought I should get
>>
>>>>> xx.view([('',xx.dtype)]*xx.shape[1])
>> array([[(0.0, 0.0, 0.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)],
>>       [(1.0, 1.0, 1.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)],
>>       [(2.0, 2.0, 2.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)]],
>>      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])
>>
>> if I make a copy and then construct a view as structured array, I get
>> what I want
>>
>>>>> xx2 = xx.copy()
>>>>> xx2.view([('',xx2.dtype)]*xx2.shape[1])
>> array([[(0.0, 0.0, 1.0)],
>>       [(0.0, 1.0, 1.0)],
>>       [(0.0, 2.0, 1.0)],
>>       [(1.0, 0.0, 1.0)],
>>       [(1.0, 1.0, 1.0)],
>>       [(1.0, 2.0, 1.0)],
>>       [(2.0, 0.0, 1.0)],
>>       [(2.0, 1.0, 1.0)],
>>       [(2.0, 2.0, 1.0)]],
>>      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])
>>
>> Are there rules for this behavior or a description in the docs,
>> because my mistakes in this are quite difficult to debug? Or do I have
>> to make a copy by default as in matlab?
>
> .view() makes a view onto the memory, not onto the
> memory-as-if-it-were-normalized-to-C-contiguous-order. Use
> ascontiguous() if you need to ensure the latter.
>

I guess that's   np.ascontiguousarray(xx)

Thanks, I'm still trying to get the trickier parts of numpy.

I was trying this out some more in order to understand the view
examples. In my other examples, numpy raises an exception of the array
is not c-contiguous when constructing a view, see below.

Shouldn't numpy raise an exception also for my original example, since
also there the array wasn't c-contiguous instead of silently using the
array base?

At least it would reduce debugging time, since I am not used to having
to think about memory layout when I just do some array and matrix
calculations and the arrays are not what they appear to be. (coming
from other, more restrictive matrix languages like matlab or gauss).

Related to my original problem, is there a way to convert between
structured arrays and plain ndarrays (in both directions) that forces
a copy instead of just creating a view?

Josef

Josef


>>> import numpy as np

>>> xx = np.array([[ 0.,  0.,  1.],
...        [ 0.,  1.,  1.],
...        [ 0.,  2.,  1.],
...        [ 1.,  0.,  1.],
...        [ 1.,  1.,  1.],
...        [ 1.,  2.,  1.],
...        [ 2.,  0.,  1.],
...        [ 2.,  1.,  1.]])

>>> xx.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> xx.view([('',float),('',float),('',float)])
array([[(0.0, 0.0, 1.0)],
       [(0.0, 1.0, 1.0)],
       [(0.0, 2.0, 1.0)],
       [(1.0, 0.0, 1.0)],
       [(1.0, 1.0, 1.0)],
       [(1.0, 2.0, 1.0)],
       [(2.0, 0.0, 1.0)],
       [(2.0, 1.0, 1.0)]],
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])
>>> xx[:6,:].flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> xx[:6,:].view([('',float),('',float),('',float)])
array([[(0.0, 0.0, 1.0)],
       [(0.0, 1.0, 1.0)],
       [(0.0, 2.0, 1.0)],
       [(1.0, 0.0, 1.0)],
       [(1.0, 1.0, 1.0)],
       [(1.0, 2.0, 1.0)]],
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])


but the following raise exceptions as desired

>>> xx[:,:2].flags
  C_CONTIGUOUS : False
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> xx[:,:2].view([('',float),('',float)])

Traceback (most recent call last):
  File "<pyshell#326>", line 1, in <module>
    xx[:,:2].view([('',float),('',float)])
ValueError: new type not compatible with array.


>>> xx.T[:,:2].view([('',float),('',float)])

Traceback (most recent call last):
  File "<pyshell#327>", line 1, in <module>
    xx.T[:,:2].view([('',float),('',float)])
ValueError: new type not compatible with array.
>>> xx.T[:,:2].flags
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False


the following also produces a view on the base, not the one that I
expected, and no exception is raised

xx.reshape(-1,2,order='F').view([('',float),('',float)])