low-rank optimization

3 papers

• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
• Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training