gradient clipping
A technique used during optimization to prevent gradients from becoming too large, which can lead to instability in learning, especially in deep neural networks.
- Adaptive Sigmoid Clipping for Balancing the Direction–Magnitude Mismatch Trade-off in Differentially Private Learning
- Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions
- Escaping saddle points without Lipschitz smoothness: the power of nonlinear preconditioning
- GeoClip: Geometry-Aware Clipping for Differentially Private SGD
- Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
- Tight High-Probability Bounds for Nonconvex Heavy-Tailed Scenario under Weaker Assumptions