data parallelism

3 papers

• Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
• MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
• Power Lines: Scaling laws for weight decay and batch size in LLM pre-training