Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

Robert Sim (Microsoft Corporation) · Yehya Farhat (Rice University) · Hamza ElMokhtar Shili (Rice University) · Fangshuo Liao (Rice University) · Chen Dun (Rice University) · Mirian Hipolito Garcia (Microsoft) · Guoqing Zheng (Percepta AI) · Ahmed Awadallah (Microsoft) · Dimitrios Dimitriadis (Microsoft Research) · Anastasios Kyrillidis (Rice University)

accuracy improvement data heterogeneityddomedecentralized training distributional shifts domain-specific expertisedynamically decentralized orchestrationexpert specialization federated learning gating mechanisms joint training mixture-of-expertspersonalized expert selectiontheoretical insights zero-shot generalization

Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing *Dynamically Decentralized Orchestration of MoEs* or *DDOME*. *DDOME* leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, *DDOME* achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate *DDOME* within a Federated Learning (FL) context: *DDOME* attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.