learning rate
A hyperparameter that determines the size of the steps taken during the optimization process as the model learns. It influences how quickly or slowly a model converges to a solution and can significantly affect training stability and performance.
- AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares
- Does Stochastic Gradient really succeed for bandits?
- Does Stochastic Gradient really succeed for bandits?
- From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGD
- Gemstones: A Model Suite for Multi-Faceted Scaling Laws
- How to Scale Second-Order Optimization
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful
- The Primacy of Magnitude in Low-Rank Adaptation