Bias and Variance

machine-learningbias-variance-tradeoffoverfittingmodel-complexitystatistics

Abstraction: Bias-variance decomposition explains model overfitting and underfitting tradeoff

Key points:

Prediction error decomposes as: Err(x) = Bias² + Variance + Irreducible Error; only bias and variance are reducible
Bias = systematic offset of expected predictions from truth; variance = spread of predictions across different training sets
Illustrated with k-NN: small k → high variance (jagged boundaries, islands), large k → high bias (smooth but misaligned boundary); increasing k decreases variance and increases bias
Analytical k-NN formula: Variance term = σ²/k (falls as k increases); Bias term depends on local roughness of the true model
Bagging (Bootstrap Aggregating) reduces variance without increasing bias; Random Forests exploits this — averaging many high-variance decision trees yields low-variance ensemble
Sweet spot is where dBias/dComplexity = −dVariance/dComplexity; found empirically via cross-validation, not analytically

Connections: Bias Variance Tradeoff · Overfitting · Model Complexity · K Nearest Neighbors