[Numpy-discussion] Ransom Proposals

Tim Hochberg tim.hochberg at cox.net
Tue Mar 28 14:40:04 CST 2006


Travis Oliphant wrote:

> Tim Hochberg wrote:
>
>>
>> +1 Assuming I understand all of the implications.
>>
>> * When doing automated conversion from Numeric/numarray to numpy, 
>> "reshape(obj) => asarray(obj).reshape()". Doing anything else is 
>> going to cause breakage and/or massive slowdowns. Can the conversion 
>> tool handle this? I would guess yes, but that's just a guess.
>
> Why would we need to change this --- even to 
> ascontiguousarray(obj).reshape()?

Because things could break or get very very slow. Perhaps I'm the only 
one who's ever used reshape to excess, so perhaps it won't really hurt 
anyone. You can see the code that originally inspired my loathing for 
reshape here:
    http://starship.python.net/~hochberg/fourier.py.html
Look at fft_base2. I don't *think* that this code would break under this 
change,  however it would certainly start running very slow if reshape 
started returning copies all the time. On the other hand, 
asarray(x).reshape would work about the same.

It is true that I have nothing comparable in my active codebase, so 
maybe such stuff is too rare to worry about. [The code above is quite 
old and may not even work on current Numeric. It was quite cool though 
-- running within a factor of two in speed of Numeric's built in FFT for 
large power of two length arrays].

So in summary, some things could break. Or they could become really 
slow. I don't know how much of a problem, if any, that this will be in 
practice. I didn't find anything in my active codebase where it would be 
a problem, so I guess it doesn't matter to me.

>>
>> * This means that "somenoncontiguousarray.reshape(newshape)" will 
>> oftern raise an error. That's a good thing, but I figured I'd get it 
>> out in the open now so no one is suprised.
>
> I'm still not sold that raising an error here is a good thing, which 
> is why I think the function should work as it always as and *not* be 
> deprecated or hard to get at even if the method is changed to raise an 
> error if the reshape cannot be accomplished without a copy.

I'm not sure what you are saying here. My bullet point above only 
relates to methods. It sounds like your on board with methods always 
returning views. Yet you disagree. Are we talking past each other here?

> Discontiguous arrays actually show up quite a bit.  I'm not convinced 
> that Tim's concern over "confusion" between when something returns a 
> view and when it returns a copy is that big of a deal.  I maintain 
> it's only disconcerting for someone who wants to do something 
> "advanced".  That person will know how to figure it out.   If things 
> are "fixed" then new users will get bitten by slow code or errors that 
> don't make sense.  I'm opposed to making things unnecessarily hard for 
> new users.

Me too. But I have come to opposite conclusion from you here. If the 
functions always returned a copy, they are safe and easy to use. And the 
error message for the method reshape could be made quite informative:

"""cannot reshape a discontiguous array with the method reshape
     use the function reshape instead or use ascontiguousarray to get a 
contiguous array"""

With something like that in place, I think it would be hard to get stuck 
to too long if one ran into an error reshaping a discontiguous array.

>
> Let's discuss this with a particular example in mind that is not 
> contrived to do 'in-place' operations.   I think this will exemplify 
> the crux of the matter for those who may not quite understand what 
> this discussion is about.
>
> a = rand(4,5,3)
> b = a.transpose().reshape(15,4)
> c = a.reshape(15,4)
>
> In this example b is now a copy of the data in a and c is a "view" of 
> the data in a.   The reason for this is because a.transpose() returns 
> a "view" of the data which is not C-style contiguous.   The reshape 
> function needs to know how to go between a linear sequence of elements 
> and a multidimensional array --- the default is current C-style 
> contiguous (last-index varies the fastest).
> If I understand correctly, Tim want's the second line to raise an 
> error so that the reshape method *always* returns a view.  

Yes. Well that's one of my many positions. I've stated my primary goal 
many times, which is that with a few careful exceptions, functions 
should always return views or always return copys. There are many ways 
to get closer to that goal, but each one has someone who dislikes it. 
This approach is one of those ways.

> I'm not enthused about having the second line raise an error because 
> it is an example of common code (perhaps a transpose does not precede 
> the reshape but a slice selection --- there are lots of ways to get 
> discontiguous arrays).   As it stands, I think the only problems that 
> arise are when the user wants to modify b and/or c in-place.  Then, 
> one is not sure whether or not b (or c)  is pointing to the data of a 
> or not. 
> My argument is that the reshape method documentation should state that 
> you should not modify the contents of the return value of reshape 
> unless it doesn't matter whether or not you have a copy of the data.  
> I would even be willing to make 'views' un-writeable by default so 
> that there was no way to "inadvertently" modify the data in a, unless 
> you know you want to.

I weeble and wobble on this suggestion. It is appealing, but it is a 
little weird. To make it truly safe you'd need to lock both the operand 
and the result, which I expect would make people squawk..

>
>
> We had this discussion for ravel months ago and the flatten method 
> which always returns copies was created.   The ravel method is another 
> one that returns views when it can (contiguous) and copies when it can't.
> I think reshape and ravel are the only methods that behave this way.   
> I'm not convinced that changing their current, standard behavior is 
> going to buy us anything but more confusion.

Wow!  So we have:

    reshape(-1)
    ravel()
    flatten()
    flat

And that's just the methods. That's not on todays agenda however.

>>
>> * Since x.reshape would no longer return copies, the order flag is 
>> unnecessary.
>
> I don't believe this.  See my other posts on the difference between 
> the array striding and how you want to interpret the array.

I think I see the root of our difference here. My thinking has been that 
the underlying data could be stored in either Fortran-order or C-order, 
but the numarray objects themselves were always C-order in the sense 
that the last axes were viewed as varying fastest. Thus you only need to 
know the order when you copy it since that controls the layout of the 
underlying data structure.

On the other hand, your view is that the entire numarray object can be 
viewed as either C-order or Fortran-order. Thus, every reshaping 
operation needs to know the order of the matrix.

Is that correct?

I'm not sure how I feel about that, but at least I think I've got it now.

And if we view the arrays that way, doesn't that mean flat is broken?

>>
>> * numpy.reshape(obj, newshape) on the other hand, always copies, so 
>> it *could* grow an order flag. I think that I'd prefer that it 
>> didn't; if you need that kind of control of the resulting arrays you 
>> should be using array to copy your matrices.
>
> No, this is not what I understand.  the reshape function would not 
> *always* copy but would behave as it currently does.  Again, you 
> haven't sold me on the need to absolutely, all the time, return either 
> views or copies.    I think such a dogmatic view is too limiting in 
> this case.
>
> I have not been bitten by this in over 10 years.  Why is this such an 
> issue now?

Because it's probably the only chance we'll ever get to change some of 
these things.

>>
>> * What does fixing ravel to be consistent mean? Always return a view? 
>> Or always return a copy? The latter seems more useful, as the first 
>> is already covered by both obj.reshape(-1) and obj.flat.
>
>
> I meant that whatever the behavior is for .reshape should be the same 
> for .ravel since it is syntactic sugar for obj.reshape(-1)
>
> And obj.flat is not even in the same ball-park since it returns an 
> array_iterator object and not an array.
>
Regards,

-tim





More information about the Numpy-discussion mailing list