adaptive optimizers

4 papers

• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
• Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
• Gradient Multi-Normalization for Efficient LLM Training
• Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis