Ridge regression - Wikipedia

Abstraction: L2 regularization adding penalty to stabilize ill-conditioned regression

Key points:

• Adds λI to the moment matrix X^TX in OLS, reducing condition number and mitigating multicollinearity; estimator is (X^TX + λI)^{-1}X^Ty
• Introduced by Hoerl and Kennard in 1970 in Technometrics; named after Andrey Tikhonov who independently developed it for integral equations
• In ML called weight decay; also known as L2 regularization, Tikhonov-Phillips regularization, constrained linear inversion
• Bayesian interpretation: equivalent to MAP estimation with a zero-mean Gaussian prior on coefficients
• Ridge parameter λ chosen via cross-validation or Grace Wahba's generalized cross-validation minimizer
• SVD analysis shows λ adds to singular values in denominator, shrinking small singular value contributions

Connections: Andrey Tikhonov · Ridge Regression · Regularization · Bias Variance Tradeoff

Source: http://en.wikipedia.org/wiki/Tikhonov_regularization