attention patterns
Refers to the way models, particularly those using attention mechanisms, focus on different inputs selectively, allowing them to weigh the importance of various features in the data.
- $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
- Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
- Pinpointing Attention-Causal Communication in Language Models
- Rope to Nope and Back Again: A New Hybrid Attention Strategy
- SPARTAN: A Sparse Transformer World Model Attending to What Matters