Gradient descent for linear regression¶

Abstraction: GD, SGD, and minibatch SGD implementations for least-squares linear regression

Key points:

• Least squares has an analytical solution (w = (X^T X)^-1 X^T y), but gradient descent is preferred when N or features are large, or for regularized variants (Ridge/Lasso)
• Three implementations shown: batch GD (full dataset, tolerance-based early stopping), SGD (one sample at a time; Widrow-Hoff), minibatch SGD (common for GPU training)
• Gradient w.r.t. weights: gradient_w = 2 error x / N; update: w -= eta * gradient_w
• Learning rate (eta) critically affects convergence: too low = slow; too high = divergence or oscillation
• Minibatch SGD is standard for neural networks on GPUs — balances parallelism with memory constraints
• All three variants converge to similar MSE on test set for this simple univariate toy problem

Connections: Scikit Learn · Gradient Descent · Stochastic Gradient Descent · Linear Regression · Optimization

Source: https://www.cse.chalmers.se/~richajo/dit866/backup_2019/lectures/l5/GD%20and%20SGD%20regression%20example.html