[Numpy-discussion] Slice assignment and overlapping views (was: Strange behavior in setting masked array values in Numpy 1.1.0)

Pauli Virtanen pav@iki...
Sun Jun 1 03:29:15 CDT 2008


su, 2008-06-01 kello 03:37 -0400, Anne Archibald kirjoitti:
> 2008/5/31 Pauli Virtanen <pav@iki.fi>:
> 
> > The reason for the strange behavior of slice assignment is that when
> the
> > left and right sides in a slice assignment are overlapping views of
> the
> > same array, the result is currently effectively undefined. Same is
> true
> > for ndarrays:
> >
> >>>> import numpy
> >>>> a = numpy.array([1, 2, 3, 4, 5])
> >>>> a[::-1]
> > array([5, 4, 3, 2, 1])
> >>>> a[:] = a[::-1]
> >>>> a
> > array([5, 4, 3, 4, 5])
> 
> I think that the current rule is, slices are walked from low index to
> high index. This doesn't help with multidimensional arrays, where the
> order of the axes is (and should be) determined by efficiency
> considerations.

Yes, I figured that it is not possible to do what the user usually means
in all cases without using at least one scalar temporary.

[clip]
> Unfortunately there's really no good way to warn about overlapping
> copies. Remember that this is a frequent operation, so it has to be
> fast for small arrays.

I didn't actually mean detecting overlaps, but detecting when assigning
to a view:

        while (a->base) a = a->base;
        while (b->base) b = b->base;
        if (a == b) {
            raise warning, but continue
        }

The leading loops can be removed if the bases are walked upwards on
array creation, and in this case an added single branch cannot be so bad
of an performance hit. If the branch is taken, then there's a
performance hit of course.

> I think changing base so that it points to the real base and not the
> parent would help (and clear up a memory leak: try "while True: A =
> A[::-1]" some time) eliminate some cases where overlap cannot occur,
> but what about the following cases?
> 
> A[:5] = A[-5:]
> A[::2] = A[1::2]
> A[1:] = A[-1:]
> 
> The last is actually fairly common (I've needed it), and relies on
> numpy's ordering of copies. The middle one is very common, and the
> first one would be a royal pain to code around if the slices were not
> allowed to overlap.

But maybe the warning is still too much hand-holding, and we can just
add a warning about undefined behavior in the documentation and leave it
at that. I can't argue much against your examples.

If the behavior in 1d for overlapping parts is well-defined and relied
upon, we should add a unit test that ensures it, if there isn't one yet
(didn't check yet).

[clip]
> In any case, many users need nearly-overlapping slices, and some need
> really-overlapping slices. Preventing problems is going to have to
> happen at a higher level.

"Higher level" means "Documentation" here, right?

	Pauli




More information about the Numpy-discussion mailing list