mixture of experts
This architecture consists of multiple specialized models (experts) that are trained to handle different subsets of data, with a gating mechanism to determine which expert is most relevant for a particular input, thereby improving efficiency and performance.
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
- MoBA: Mixture of Block Attention for Long-Context LLMs
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
- PINN Balls: Scaling Second-Order Methods for PINNs with Domain Decomposition and Adaptive Sampling
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
- SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling