[Numpy-discussion] calculating the mean and variance of a large float vector
Bruce Southey
bsouthey@gmail....
Fri Jun 6 09:16:32 CDT 2008
Bruce Southey wrote:
> Alan McIntyre wrote:
>> On Thu, Jun 5, 2008 at 10:16 PM, Keith Goodman <kwgoodman@gmail.com>
>> wrote:
>>
>>> How can that lead to instability? If the last half-million values are
>>> small then they won't have a big impact on the mean even if they are
>>> ignored. The variance is a mean too (of the squares), so it should be
>>> stable too. Or am I, once again, missing the point?
>>>
>>
>> No, I just didn't think about it long enough, and I shouldn't have
>> tried to make an example off the cuff. ;) After thinking about it
>> again, I think some loss of accuracy is probably the worst that can
>> happen.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion@scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> Any problems are going to mainly due to the distribution of numbers
> especially if there are very small numbers and very large numbers.
> This is mitigated by numerical precision and algorithm - my guess is
> that it will take a rather extreme case to cause you any problems.
>
> Python and NumPy are already using high numerical precision (may
> depend on architecture) and NumPy defines 32-bit, 64-bit and 128-bit
> precision if you want to go higher (or lower). This means that
> calculations are rather insensitive to numbers used so typically there
> is no reason for any concern (ignoring the old Pentium FDIV bug,
> http://en.wikipedia.org/wiki/Pentium_FDIV_bug ).
>
> The second issue is the algorithm where you need to balance
> performance with precision. For simple calculations:
> http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
>
> Bruce
>
I forgot to add:
import numpy
x = 1e305 * numpy.ones(10000000, np.float128)
type(x[0]) # gives <type 'numpy.float128'>
x.mean() # gives 1.000000000000036542e+305
Bruce
More information about the Numpy-discussion
mailing list