[Numpy-discussion] precision isn't just precision
Paul F. Dubois
pauldubois at home.com
Tue Feb 8 11:56:11 CST 2000
Before we all rattle on too long about precision, I'd like to point out that
selecting a precision actually carries two consequences in the context of
computer languages:
1. Expected: The number of digits of accuracy in the representation of a
floating point number.
2. Unexpected: The range of numbers that can be represented by this type.
Thus, to a scientist it is perfectly logical that if d is a double and f is
a single,
d * f
has only single precision validity.
Unfortunately in a computer if you hold this answer in a single, then it may
fail if the contents of d include numbers outside the single range, even if
f is 1.0.
Thus the rules in C and Fortran that coercion is UP had to do as much with
range as precision.
More information about the Numpy-discussion
mailing list