Gradient Descent

The Workhorse of Machine Learning Optimization

Mathematical Foundation

Understand gradients and the optimization landscape

Batch vs Stochastic

Master different gradient descent variants

Mini-batch Strategy

Learn the practical compromise approach

Advanced Methods

Explore momentum and adaptive learning rates

What is Gradient Descent?

Gradient descent is an iterative optimization algorithm that finds the minimum of a function by following the direction of steepest descent, as indicated by the negative gradient.

Core Intuition

  • Imagine rolling a ball down a hill
  • Ball naturally follows steepest descent
  • Eventually reaches the bottom (minimum)
  • Gradient points uphill, so we go opposite direction
Real-world Impact: Gradient descent trains virtually every neural network, from simple perceptrons to GPT and other large language models.
3D Optimization Landscape

Mathematical Foundation

The Gradient Vector

$$\nabla f(\boldsymbol{\theta}) = \begin{bmatrix} \frac{\partial f}{\partial \theta_1} \\ \frac{\partial f}{\partial \theta_2} \\ \vdots \\ \frac{\partial f}{\partial \theta_n} \end{bmatrix}$$

The gradient points in the direction of steepest increase of the function.

Update Rule

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \nabla f(\boldsymbol{\theta}_t)$$
  • θ: Parameters to optimize
  • α: Learning rate
  • ∇f: Gradient of objective function

Key Insight: We move in the direction opposite to the gradient because we want to minimize the function, not maximize it.

Learning Rate: The Critical Hyperparameter

Learning Rate Effects Visualization

Too Small (α ≪ 1)

  • Very slow convergence
  • Many iterations needed
  • Safe but inefficient
  • May get stuck in plateaus

Just Right (α ≈ optimal)

  • Smooth, fast convergence
  • Efficient optimization
  • Reaches minimum quickly
  • Problem-dependent value

Too Large (α ≫ 1)

  • Oscillating behavior
  • May overshoot minimum
  • Potential divergence
  • Unstable optimization

Batch Gradient Descent

Algorithm: Batch Gradient Descent

Compute gradient using entire training dataset at each step

$$\nabla f(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \nabla f_i(\boldsymbol{\theta})$$

Advantages

  • Guaranteed convergence for convex functions
  • Smooth convergence path
  • Stable gradient estimates
  • Theoretical guarantees

Disadvantages

  • Computationally expensive per iteration
  • Memory intensive for large datasets
  • Slow for big data applications
  • May get stuck in local minima
When to Use: Small to medium datasets, when you need precise convergence, or when computational resources are abundant.

Stochastic Gradient Descent (SGD)

Algorithm: Stochastic Gradient Descent

Use gradient from single randomly selected sample at each step

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \nabla f_i(\boldsymbol{\theta}_t)$$

Advantages

  • Fast iterations (one sample per update)
  • Low memory requirements
  • Can escape local minima due to noise
  • Online learning capability
  • Scales to massive datasets

Disadvantages

  • Noisy convergence path
  • May not converge to exact minimum
  • Requires learning rate decay
  • Sensitive to hyperparameters
SGD Convergence Path (Noisy)

Mini-batch Gradient Descent

Mini-batch gradient descent combines the best of both worlds: computational efficiency of SGD with stability of batch gradient descent.

$$\nabla f(\boldsymbol{\theta}) = \frac{1}{m}\sum_{i \in \mathcal{B}} \nabla f_i(\boldsymbol{\theta})$$

Batch Size Selection

  • Small batches (32-128): More noise, faster iterations
  • Medium batches (256-512): Good balance for most problems
  • Large batches (1024+): More stable, requires more memory
Rule of Thumb: Start with batch size 32-256, increase if you have computational resources and want stability.
Batch Size Effects Comparison

Gradient Descent Variants Comparison

Aspect
Batch GD
SGD
Computation per Iteration
O(n) - entire dataset
O(1) - single sample
Memory Usage
High - stores all gradients
Low - one sample at a time
Convergence
Smooth and stable
Noisy but can escape local minima
Parallelization
✓ Easy to parallelize
✗ Sequential by nature
Large Datasets
✗ Impractical
✓ Excellent scalability
Online Learning
✗ Not suitable
✓ Perfect fit
Mini-batch GD: Offers a practical compromise with batch sizes typically between 32-512, combining efficiency with stability.

Advanced Gradient Descent Methods

SGD with Momentum

$$\mathbf{v}_t = \beta \mathbf{v}_{t-1} + \alpha \nabla f(\boldsymbol{\theta}_t)$$ $$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \mathbf{v}_t$$

Accumulates velocity to accelerate in consistent directions and dampen oscillations.

Adam Optimizer

$$\mathbf{m}_t = \beta_1 \mathbf{m}_{t-1} + (1-\beta_1)\nabla f(\boldsymbol{\theta}_t)$$ $$\mathbf{v}_t = \beta_2 \mathbf{v}_{t-1} + (1-\beta_2)(\nabla f(\boldsymbol{\theta}_t))^2$$

Combines momentum with adaptive learning rates for each parameter.

When to Use Each Method

  • Vanilla SGD: Simple problems, when you want full control
  • SGD + Momentum: Training deep networks, accelerated convergence
  • Adam: Default choice for most deep learning applications
  • RMSprop: RNNs and problems with sparse gradients

Practical Advice

Start with Adam (α=0.001) for most problems. Use SGD+momentum for fine-tuning or when you need more control.

Key Takeaways

Fundamental Principles

  • Gradient descent follows the steepest descent direction
  • Learning rate controls step size and convergence
  • Batch size affects stability vs. computational efficiency
  • Different variants suit different problem scales

Practical Guidelines

  • Use mini-batch GD for most practical applications
  • Start with Adam optimizer for deep learning
  • Tune learning rate carefully - often most important hyperparameter
  • Consider momentum for faster convergence

Gradient descent is the foundation of machine learning optimization. Understanding its variants helps you choose the right tool for your specific problem constraints and computational resources.

Modern Reality: While frameworks automate implementation, understanding these principles helps you debug convergence issues, select appropriate optimizers, and tune hyperparameters effectively.
1 / 10