[SciPy-user] fft numerical precision

Robert Kern robert.kern at gmail.com
Wed Dec 21 14:03:46 CST 2005

Ryan Krauss wrote:
> I have run into this as well.  1e-16 seems large to me for double
> precision.  Robert can you explain this a bit more please.  Is there
> an IEEE spec or something that specifies how much of the 64bits is for
> exponent and how much is for the mantissa (I think that is the right
> word)?   I was playing with some FORTRAN code and it seems like there
> was a big difference with complex vs. double complex.  It seems like
> 1e-16 was the magnitude floor of complex and double complex was
> better.

IEEE-754 double precision has 11 bits for exponent, 1 bit for sign, and 52 for
the rest of the mantissa. 2.**-52 ~= 2.22e-16

Goldberg, David. What Every Computer Scientist Should Know About Floating Point
Arithmetic. 1991.

Robert Kern
robert.kern at gmail.com

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

More information about the SciPy-user mailing list