memory-efficient training

3 papers

• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
• Gradient Multi-Normalization for Efficient LLM Training
• Irrational Complex Rotations Empower Low-bit Optimizers