decentralized training
- Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
- Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
- Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
- Unveiling the Power of Multiple Gossip Steps: A Stability-Based Generalization Analysis in Decentralized Training