learning rates
Typically refers to the setting of multiple learning rates for different layers or components in a neural network, which can enhance training efficiency and performance tuning over standard single-rate approaches.
- CoT Information: Improved Sample Complexity under Chain-of-Thought Supervision
- Comparing Uniform Price and Discriminatory Multi-Unit Auctions through Regret Minimization
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- In Search of Adam’s Secret Sauce
- In Search of Adam’s Secret Sauce
- Marginal-Nonuniform PAC Learnability
- Model Merging in Pre-training of Large Language Models
- Online Inverse Linear Optimization: Efficient Logarithmic-Regret Algorithm, Robustness to Suboptimality, and Lower Bound
- Pay Attention to Small Weights
- Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability