batch size
Batch size is the number of training examples utilized in one iteration of model training, influencing convergence speed, gradient estimation stability, and memory requirements.
- Compute-Optimal Scaling for Value-Based Deep RL
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
- Rethinking Residual Distribution in Locate-then-Edit Model Editing
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful
- Understanding outer learning rates in Local SGD
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks