[SciPy-User] How to estimate error in polynomial coefficients from scipy.polyfit?

Anne Archibald peridot.faceted@gmail....
Sun Mar 28 18:46:29 CDT 2010


On 25 March 2010 23:40, David Goldsmith <d.l.goldsmith@gmail.com> wrote:
> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin@gmail.com> wrote:
>>
>> Yikes!  This sounds like it may be more trouble than it's worth.  I
>> have a few sets of statistical data that each need to have curves fit
>> to them.
>
> That's an awfully generic need - it may be obvious from examination of the
> data that a line is inappropriate, but besides polynomials there are many
> other non-linear models (which can be linearly fit to data by means of data
> transformation) which possess fewer parameters (and thus are simpler from a
> parameter analysis perspective).  So, the question is: why are you fitting
> to polynomials?  If it's just to get a good fit to the data, you might be
> getting "more fit" than your data warrants (and even if that isn't a
> problem, you probably want to use a polynomial form different from "standard
> form," e.g., Lagrange interpolators).  Are you sure something like an
> exponential growth/decay or power law model (both of which are "more
> natural," linearizable, two-parameter models) wouldn't be more appropriate -
> it would almost certainly be simpler to analyze (and perhaps easier to
> justify to a referee).
>
> On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?  The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.

In pulsar timing, what you measure is phase (actually pulse arrival
time); pulse period/frequency is its derivative, and spin-down is the
derivative of period and measures the power available for producing
radiation. These two parameters are also used in standard formulas to
infer magnetic field and age. The second derivative of period (third
derivative of the measured quantity) is generally too small to be
measured, but when it can be it tells you something about the
mechanism by which the spin-down is happening.

Beyond the plain physics, there is typically some low-frequency noise
(nobody's sure where it comes from, possibly the turbulence in the
superfluid layers of the neutron star) which is typically managed by
subtracting a best-fit polynomial (polynomial subtraction has the
property of killing off noise spectra that diverge at zero frequency,
which this noise appears to).

I suppose power laws, though ubiquitous, do not exactly count as
polynomials; generally their exponents are not integers.

Anne

> DG
>
>>
>> I can see what the parameters are and they are similar
>> between each set.  I want to know if the coefficients are within a few
>> standard deviations of each other.  That's what I was hoping to get
>> at, but it seems to be more trouble than I'm willing to accept.
>> Unless anyone has a simpler solution.
>>
>> Thanks,
>> Jeremy
>> >
>> > Anne
>> >
>> >> residuals, rank, singular_values, rcond : present only if `full` = True
>> >>        Residuals of the least-squares fit, the effective rank of the
>> >> scaled
>> >>        Vandermonde coefficient matrix, its singular values, and the
>> >> specified
>> >>        value of `rcond`. For more details, see `linalg.lstsq`.
>> >>
>> >> I don't think any of these things are "design matrix" as you have
>> >> indicated I need.  The documentation for linalg.lstsq does not say
>> >> what rcond is unfortunately.   Any ideas?
>> >>
>> >> Jeremy
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User@scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >>
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


More information about the SciPy-User mailing list