weight decay
A regularization technique used in training neural networks to prevent overfitting by adding a penalty to the loss function based on the magnitude of the weights. This encourages simpler models that generalize better on unseen data.
- A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation
- AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
- Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets
- Finite-Time Bounds for Average-Reward Fitted Q-Iteration
- Generalization Bounds for Rank-sparse Neural Networks
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
- How to Scale Second-Order Optimization
- Learning in Compact Spaces with Approximately Normalized Transformer
- Low Rank Gradients and Where to Find Them
- Superposition Yields Robust Neural Scaling
- Superposition Yields Robust Neural Scaling