Learning Rate Schedulers

Optimizing Optimization Through Adaptive Learning Rates

Scheduling Strategies

Master step decay, cosine, and polynomial scheduling

Adaptive Methods

Learn performance-based learning rate adjustment

Advanced Techniques

Explore warm-up, cyclical rates, and one-cycle policies

Practical Implementation

Apply scheduling strategies in real-world scenarios

Why Learning Rate Scheduling?

Static learning rates often lead to suboptimal convergence. Learning rate scheduling adapts the step size during training to achieve better optimization and faster convergence.

Problems with Fixed Learning Rates

  • Too high: oscillation around minimum
  • Too low: extremely slow convergence
  • Doesn't adapt to changing loss landscape
  • May get stuck in plateaus
  • Suboptimal final performance
Key Insight: Early training benefits from larger steps for fast progress, while later training needs smaller steps for fine-tuning.
Fixed vs Scheduled Learning Rate Comparison

Step Decay Schedulers

Step Decay

$$\alpha_t = \alpha_0 \times \gamma^{\lfloor t/s \rfloor}$$
  • α₀: Initial learning rate
  • γ: Decay factor (e.g., 0.1)
  • s: Step size (epochs between drops)

Exponential Decay

$$\alpha_t = \alpha_0 \times e^{-\lambda t}$$

Smooth continuous decay with decay rate λ.

Step vs Exponential Decay Visualization
When to Use: Step decay for established architectures with known training phases. Exponential for smoother transitions.

Cosine and Polynomial Schedulers

Cosine Annealing

$$\alpha_t = \alpha_{min} + \frac{\alpha_{max} - \alpha_{min}}{2}\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$$

Smooth sinusoidal decay from αₘₐₓ to αₘᵢₙ over T epochs.

Polynomial Decay

$$\alpha_t = \alpha_0 \times \left(1 - \frac{t}{T}\right)^p$$

Power law decay with exponent p controlling decay rate.

Cosine vs Polynomial Decay Curves
Cosine Benefits: Smooth transitions, no hyperparameter tuning for decay rate. Works well with warm restarts.

Warm Restarts and Cyclical Learning Rates

Cyclical learning rates periodically restart the learning rate to escape local minima and explore different regions of the loss landscape.

Cosine Annealing with Warm Restarts

$$\alpha_t = \alpha_{min} + \frac{\alpha_{max} - \alpha_{min}}{2}\left(1 + \cos\left(\frac{T_{cur}}{T_i}\pi\right)\right)$$

Where T_cur is epochs since last restart, T_i is restart period.

Benefits of Restarts

  • Escape local minima
  • Explore multiple solutions
  • Often find better final solutions
  • Ensemble-like behavior
Warm Restarts Visualization

Adaptive Learning Rate Schedulers

ReduceLROnPlateau

  • Monitor validation loss (or other metric)
  • Reduce LR when no improvement for N epochs
  • Typically reduce by factor of 0.1-0.5
  • Automatically adapts to training dynamics
Parameters:
  • Patience: epochs to wait
  • Factor: reduction multiplier
  • Min LR: lower bound

Learning Rate Range Test

  • Gradually increase LR from very small to large
  • Plot loss vs learning rate
  • Find optimal range from curve shape
  • Use steepest descent point
LR Range Test Curve

One-Cycle Learning Rate Policy

One-cycle policy: start low, increase to maximum, then decrease to very low. Achieves super-convergence with fewer epochs and better generalization.

One-Cycle Learning Rate Policy

Phase Breakdown

  • Phase 1 (45%): Increase from base to max LR
  • Phase 2 (45%): Decrease from max to base LR
  • Phase 3 (10%): Decrease to minimum LR

Benefits

  • Faster convergence (super-convergence)
  • Better generalization
  • Requires fewer epochs
  • Self-regularizing effect

Learning Rate Warm-up

Why Warm-up?

  • Large models sensitive to initial LR
  • Prevents early instability
  • Allows higher maximum learning rates
  • Essential for large batch training

Linear Warm-up

$$\alpha_t = \alpha_{target} \times \frac{t}{t_{warmup}}$$

Linearly increase from 0 to target LR over warm-up period.

Warm-up + Cosine Decay Schedule
Typical Setup: 5-10% of total epochs for warm-up, followed by cosine decay or other schedule.

Implementation and Best Practices

Method
Best For
Key Parameters
Step Decay
Established architectures
Step size, decay factor
Cosine
General purpose, smooth decay
T_max, min LR
ReduceLROnPlateau
Unknown training dynamics
Patience, factor
One-Cycle
Fast training, good generalization
Max LR, cycle length

Implementation Tips

  • Start with cosine annealing for most cases
  • Use LR range test to find optimal bounds
  • Monitor both training and validation metrics
  • Log learning rate for debugging

Common Pitfalls

  • Too aggressive decay (undershooting optimum)
  • Not using warm-up with large models
  • Ignoring validation performance
  • One-size-fits-all approach

Key Takeaways

Core Principles

  • Learning rate scheduling improves convergence
  • Different schedules suit different problem types
  • Adaptive methods reduce hyperparameter tuning
  • Cyclical rates can escape local minima

Practical Guidelines

  • Use cosine annealing as default choice
  • Add warm-up for large models or batch sizes
  • Try one-cycle for faster training
  • Monitor learning rate during training

Learning rate scheduling is one of the most impactful hyperparameter choices in deep learning. Proper scheduling can reduce training time, improve final performance, and enhance generalization.

Modern Practice: Most state-of-the-art models use sophisticated scheduling combining warm-up, cosine decay, and sometimes cyclical elements for optimal performance.
1 / 10