softmax attention
An attention mechanism used in transformer models where the scores for different inputs are normalized using the softmax function, allowing the model to weigh the importance of different parts of the input dynamically.
- Benign Overfitting in Single-Head Attention
- Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
- Exploring Diffusion Transformer Designs via Grafting
- Exploring Diffusion Transformer Designs via Grafting
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems
- Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
- Towards Understanding Transformers in Learning Random Walks
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers