KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction

Sangdoo Yun (Naver AI Lab) · Jang-Hyun Kim (Seoul National University) · Jinuk Kim (Seoul National University) · Sangwoo Kwon (Seoul National University) · Jae W. Lee (Seoul National University) · Hyun Oh Song (Seoul National University)

attention latencycache evictioncode comprehensioncompressed kv cachesempirical evaluations flashattention kv cache memory overheadmulti-query scenariosperformance degradationquery-agnosticquestion-answering reasoning retrieval transformer

Transformer-based large language models (LLMs) cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces \textit{KVzip}, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by $3$-$4\times$ and FlashAttention decoding latency by approximately $2\times$, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90\% cache budget ratio under multi-query scenarios.