Fitting Polynomials: Where is sloppiness from?

Abstraction: Mathematical origins of sloppiness in multiparameter polynomial models

Key points:

• Sloppiness arises when groups of parameters have similar effects on predictions and can be traded for one another without changing model output
• The Hessian of a polynomial fit over [0,1] is the Hilbert matrix — famously ill-conditioned with a huge eigenvalue ratio
• Switching from monomial coefficients (a) to orthogonal polynomial coefficients (b, e.g. shifted Legendre polynomials) eliminates sloppiness; the Hessian becomes the identity matrix
• Sloppiness is therefore a property of the parameter parameterization, not the model itself
• In scientific models, the natural "bare" parameters are almost always related to behavior-governing parameters via a severe, skewing transformation — making most models sloppy
• Three root causes: tradeable parameters, choice of parameterization, severe skewness between bare and natural parameters

Connections: James Sethna · Sloppy Models · Parameter Estimation · Hessian Matrix

Source: http://www.lassp.cornell.edu/sethna/Sloppy/FittingPolynomials.html