[Numpy-discussion] Histograms via indirect index arrays

Piotr Luszczek luszczek at cs.utk.edu
Fri Mar 17 11:26:03 CST 2006


On Friday 17 March 2006 13:29, Travis Oliphant wrote:
> Piotr Luszczek wrote:
> >>Yes, this is intended (sort of --- this particular example isn't
> >> the reason for the behavior though).
> >>
> >>The issue is that the code g[idx] +=1 is equivalent in Python to
> >>
> >>g[idx] = g[idx] + 1
> >
> >This is not what I read at:
> >
> >http://docs.python.org/ref/numeric-types.html
> >
> >Quote:
> >
> >These methods should attempt to do the operation in-place (modifying
> >self) and return the result (which could be, but does not have to
> > be, self).
> >
> >What you describe is "lack of attempt" in which case the "fall back"
> >behavior gets triggered.
>
> The problems is that this explanation is very clear when we are
> talking about the syntax
>
> g += 1
>
> Then, there is a method of g that can be over-ridden to get the
> desired behavior.  Now, what you are talking about is "indexing
> followed by in-place addition".
>
> i.e.  at issue is
>
> how does python interpret
>
> g[idx] += 1
>
> How does this get compiled to byte-code?
>
> There are two possibilities:
>
> 1) g[idx] creates a new object which then has 1 added to it using
> in-place addition.
>
>      This would not produce the desired behavior as g[idx] is a copy
> of the data when idx is a
>       general indexing array as it is in this case.  So, you make a
> copy of those indices, add 1 to them
>       and then do what with the resut?
>
> 2) g[idx] += 1  gets converted to g[idx] = g[idx] + 1
>
>     This appears to be effectively what Python actually does.  Notice
> that there is no way for us to control this behavior because there is
> no __inplace_with_indexing_add__ operator to over-ride.
>
> There is no such single operation to over-ride for the object.   In
> other words, I don't see anyay for us to even alter the object to get
> the behavior you want from that syntax.  We can, of course, add a
> function or method to do that, but I we would have to extend Python
> to get the behavior you want here.

Hardly. At least from what I'm seeing happens on a small example. 
'g[idx] += 1' becomes ('g' and 'idx' are generic objects):
__getitem__(self, idx)
__iadd__(1)
__setitem__(result of __iadd__)

By design numpy returns views from __getitem__
In this case, it would be view into 'self' and 'idx' so the __iadd__
would just use the 'idx' directly rather than a copy.
Finally, __setitem__ doesn't do anything since 'self' and 'value'
will be the same.

Of course, this is just a quick draft. I don't know how it would work
in practice and in other cases.

> Note, however, if idx is slice syntax, then the operation is done
> without making copies because, for example,
>
> g[1:10:2]
>
> returns a "view" of the data (an array that shares the same memory)
> as the original array.  Thus,
>
> g[1:10:2] += 1
>
> does an inplace add without data-copying.
>
> It may be possible to do what you want using zero-strided arrays, but
> I'll leave that for a more motivated contributor.
>
> -Travis




More information about the Numpy-discussion mailing list