second-order methods
Optimization algorithms that use information about the curvature of the loss function, generally providing faster convergence compared to first-order methods that only use gradient information.
- How to Scale Second-Order Optimization
- KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products
- Local Curvature Descent: Squeezing More Curvature out of Standard and Polyak Gradient Descent
- PINN Balls: Scaling Second-Order Methods for PINNs with Domain Decomposition and Adaptive Sampling
- Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits