hyperparameters
Hyperparameters are configuration settings used to control the training process of a model, such as learning rate, batch size, or the number of layers in a neural network. Tuning these parameters is vital for optimizing model performance.
- Accelerating Optimization via Differentiable Stopping Time
- Bag of Tricks for Inference-time Computation of LLM Reasoning
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- Compositional Reasoning with Transformers, RNNs, and Chain of Thought
- Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization
- Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
- DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
- FedRAM: Federated Reweighting and Aggregation for Multi-Task Learning
- How to Scale Second-Order Optimization
- Learning Provably Improves the Convergence of Gradient Descent
- Learning in Compact Spaces with Approximately Normalized Transformer
- Metropolis Adjusted Microcanonical Hamiltonian Monte Carlo
- Model Merging in Pre-training of Large Language Models
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
- Online robust locally differentially private learning for nonparametric regression
- Optimal Estimation of the Best Mean in Multi-Armed Bandits
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
- Random Forest Autoencoders for Guided Representation Learning
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful
- TPP-SD: Accelerating Transformer Point Process Sampling with Speculative Decoding
- Understanding outer learning rates in Local SGD
- VIKING: Deep variational inference with stochastic projections