attention sink

3 papers

• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• What are you sinking? A geometric approach on attention sink