Geometrical Interpretation of Linear Regression in Machine Learning versus Classical Statistics

linear-regressionolsgeometryregularizationstatistics

Abstraction: Geometric view of OLS, MLE, and regularization in linear regression

Key points:

OLS finds the projection of y onto the column space of X (plane colX); the error vector is perpendicular to that plane, minimizing the sum of squared perpendicular distances
Seven OLS assumptions map to geometric properties: linearity → flat plane; homoscedasticity → uniformly scattered residuals; no multicollinearity → non-collinear basis vectors; exogeneity → errors cancel out on average
MLE with normal errors yields identical coefficient estimates to OLS minimization of SSE, but differs conceptually: MLE maximizes the likelihood of observing the data, while ML optimization directly minimizes an objective
Lasso (L1) regularization can zero out coefficients (feature selection), geometrically flattening the plane in unimportant directions; Ridge (L2) shrinks all coefficients toward zero without elimination
Regularization deliberately introduces bias to reduce variance and improve generalization on unseen data — trading perfect in-sample fit for robustness

Connections: Linear Regression · Ordinary Least Squares · Regularization