gating mechanisms

6 papers

• Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
• Multi-Expert Distributionally Robust Optimization for Out-of-Distribution Generalization
• Revisiting Bi-Linear State Transitions in Recurrent Neural Networks