multi-head attention
Multi-head attention is a mechanism in neural networks, particularly in transformer architectures, that allows the model to focus on different parts of the input sequence simultaneously by using multiple attention heads, improving the model's ability to capture diverse contextual relationships in the data.
- Attention Mechanism, Max-Affine Partition, and Universal Approximation
- DAAC: Discrepancy-Aware Adaptive Contrastive Learning for Medical Time series
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
- Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
- MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
- SAS: Simulated Attention Score
- Tensor Product Attention Is All You Need