attention heads
Components of attention mechanisms in neural networks that focus on different parts of an input sequence, enabling the model to capture diverse relationships.
- A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoning
- Don’t Forget the Enjoin: FocalLoRA for Instruction Hierarchical Alignment in Large Language Models
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
- Extrapolation by Association: Length Generalization Transfer In Transformers
- Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers
- Language Models (Mostly) Know When to Stop Reading
- Lost in Transmission: When and Why LLMs Fail to Reason Globally
- Model Editing for Vision Transformers
- Proxy-SPEX: Sample-Efficient Interpretability via Sparse Feature Interactions in LLMs
- StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
- Understanding Differential Transformer Unchains Pretrained Self-Attentions
- Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models
- Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning
- When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective
- Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models