first-order methods
First-order methods are optimization algorithms that use gradient information to iteratively approach a minimum or maximum value. They are commonly used in large-scale optimization problems due to their lower computational costs compared to second-order methods.
- Accelerated Distance-adaptive Methods for Hölder Smooth and Convex Optimization
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods
- Gradient Multi-Normalization for Efficient LLM Training
- KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products
- Learning from A Single Markovian Trajectory: Optimality and Variance Reduction
- The Rich and the Simple: On the Implicit Bias of Adam and SGD