ZeroS: Zero‑Sum Linear Attention for Efficient Transformers

Jiecheng Lu (Georgia Institute of Technology) · Xu Han (Tsinghua University, Tsinghua University) · Yan Sun (New Jersey Institute of Technology) · Viresh Pati (Georgia Institute of Technology) · Yubin Kim (Georgia Institute of Technology) · Siddhartha Somani (Georgia Institute of Technology) · Shihao Yang (Georgia Institute of Technology)

additive information blendingcontrastive operationsconvex combinationsempirical benchmarkslinear attentionlong contextsmathematically stable weightsrepresentable functionsreweightingsequence modeling softmax attention transformersweight biaszero-sum linear attentionzero-sum softmax

Linear attention methods offer Transformers $O(N)$ complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term $1/t$ and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining $O(N)$ complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax attention across various sequence modeling benchmarks.