[Numpy-discussion] integer power in scalarmath: how to handle overflow?
David M. Cooke
cookedm at physics.mcmaster.ca
Mon Jun 5 16:10:23 CDT 2006
I just ran into the fact that the power function for integer types
isn't handled in scalarmath yet. I'm going to add it, but I'm wondering
what people want when power overflows the integer type?
Taking the concrete example of a = uint8(3), b = uint8(10), then should
a ** b return
1) the maximum integer for the type (255 here)
2) 0
3) upcast to the largest type that will hold it (but what if it's
larger than our largest type? Return a Python long?)
4) do the power using "long" like Python does, then downcast it to the
type (that would return 169 for the above example)
5) something else?
I'm leaning towards #3; if you do a ** 10, you get the right
answer (59049 as an int64scalar), because 'a' is upcasted to
int64scalar, since '10', a Python int, is converted to that type.
Otherwise, I would choose #1.
--
|>|\/|<
/----------------------------------------------------------------------\
|David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
More information about the Numpy-discussion
mailing list