A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

Abstraction: Survey of overparameterized ML theory explaining double descent and interpolation generalization

Key points:

• Classical dogma: overparameterized models (more parameters than training samples) that perfectly interpolate noisy data should overfit and generalize poorly
• Empirically, a wide range of interpolating models — from linear regression to deep neural networks — generalize well on unseen test data
• The double descent phenomenon shows that highly overparameterized models often outperform the best underparameterized model
• The emerging "Theory of Overparameterized ML" (TOPML) provides precise analytic characterizations of double descent in linear regression
• Understanding requires new theory even for the simplest linear model case; statistical signal processing perspective is key
• The paper surveys TOPML as a distinct subfield and outlines open research questions

Connections: Bias Variance Tradeoff · Double Descent · Overparameterization · Generalization

Source: https://arxiv.org/abs/2109.02355