Regularization Techniques

Controlling Model Complexity for Better Generalization

Bias-Variance Trade-off

Understand how regularization balances model complexity

L1 & L2 Regularization

Master Ridge and Lasso regression techniques

Modern Techniques

Explore dropout, batch norm, and data augmentation

Practical Application

Learn parameter selection and implementation strategies

What is Regularization?

Regularization is a technique to prevent overfitting by adding constraints or penalties to the model, reducing its complexity and improving generalization to new data.

The Overfitting Problem

Complex models memorize training data
Poor performance on unseen examples
High variance, low bias
Common with limited training data

Real-world Impact: A model that achieves 99% training accuracy but only 60% test accuracy is clearly overfitted and needs regularization.

Overfitting vs Regularized Model Comparison

The Bias-Variance Trade-off

$$\text{Expected Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}$$

High Bias

Underfitting
Too simple model
Poor training performance
Consistent but wrong

High Variance

Overfitting
Too complex model
Poor test performance
Inconsistent predictions

Regularization

Optimal balance
Controlled complexity
Better generalization
Stable performance

Bias-Variance Trade-off Visualization

L1 Regularization (Lasso)

Mathematical Formulation

$$\min_{\boldsymbol{\beta}} \frac{1}{2n}\|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|_2^2 + \lambda\|\boldsymbol{\beta}\|_1$$

The L1 penalty term $\lambda\|\boldsymbol{\beta}\|_1 = \lambda\sum_{j=1}^{p}|\beta_j|$ adds the sum of absolute values of coefficients.

Key Properties

Promotes sparsity (many coefficients become exactly zero)
Automatic feature selection
Non-differentiable at zero
Diamond-shaped constraint region

L1 Constraint Region (Diamond)

When to Use L1

Ideal when you suspect many features are irrelevant and want automatic feature selection.

L2 Regularization (Ridge)

Mathematical Formulation

$$\min_{\boldsymbol{\beta}} \frac{1}{2n}\|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|_2^2 + \lambda\|\boldsymbol{\beta}\|_2^2$$

The L2 penalty term $\lambda\|\boldsymbol{\beta}\|_2^2 = \lambda\sum_{j=1}^{p}\beta_j^2$ adds the sum of squared coefficients.

Key Properties

Shrinks coefficients proportionally
Handles multicollinearity well
Differentiable everywhere
Circular constraint region

L2 Constraint Region (Circle)

When to Use L2

Best when all features contribute somewhat and you want smooth coefficient shrinkage.

L1 vs L2: Geometric Intuition

Interactive L1 vs L2 Constraint Visualization

Sparsity

✓ Creates sparse solutions

✗ Dense solutions

Feature Selection

✓ Automatic

✗ Manual required

Multicollinearity

Selects arbitrarily

✓ Handles well

Computational

Iterative algorithms

✓ Closed-form solution

Elastic Net: Best of Both Worlds

$$\min_{\boldsymbol{\beta}} \frac{1}{2n}\|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|_2^2 + \lambda\left(\alpha\|\boldsymbol{\beta}\|_1 + \frac{1-\alpha}{2}\|\boldsymbol{\beta}\|_2^2\right)$$

Combining L1 and L2

α controls the mixing ratio
α = 1: Pure L1 (Lasso)
α = 0: Pure L2 (Ridge)
0 < α < 1: Elastic Net

Advantages:

Selects groups of correlated features
More stable than Lasso
Better for high-dimensional data

Elastic Net Regularization Path

When to Use Elastic Net

Ideal for high-dimensional data with grouped variables or when you need both sparsity and stability.

Modern Deep Learning Regularization

Dropout

$$y = f(\mathbf{W} \cdot (\mathbf{x} \odot \mathbf{m}))$$

Randomly sets neurons to zero during training with probability p.

Batch Normalization

$$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$$

Normalizes layer inputs, acts as implicit regularization.

Data Augmentation

Rotation, scaling, cropping for images
Noise injection for robustness
Synthetic data generation
Mixup and CutMix techniques

Early Stopping

Monitor validation loss
Stop when overfitting begins
Simple but effective
No hyperparameter tuning needed

Choosing Regularization Parameters

Cross-Validation Approach

Grid search over λ values
K-fold cross-validation
Select λ with best validation performance
Common range: 10⁻⁶ to 10²

Practical Tips:

Use logarithmic spacing for λ
Start with wide range, then narrow
Consider computational budget

Cross-Validation Curves

One Standard Error Rule: Choose the largest λ whose CV error is within one standard error of the minimum.

Key Takeaways

Fundamental Principles

Regularization prevents overfitting by controlling complexity
L1 promotes sparsity, L2 promotes smoothness
Elastic Net combines benefits of both
Modern techniques extend beyond penalty methods

Practical Guidelines

Use cross-validation for parameter selection
Consider problem characteristics when choosing technique
Combine multiple regularization approaches
Monitor both training and validation performance

Regularization is essential for building robust machine learning models. The choice of technique depends on your data characteristics, computational constraints, and interpretability requirements.

Future Directions: Research continues into adaptive regularization, learned regularization functions, and regularization for specific architectures like transformers and graph neural networks.