Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference

Jianye Hao (Tianjin University) · Jie Wang (Southeast University) · Mingxuan Yuan (Huawei Noah's Ark Lab) · Feng Wu (The University of HongKong) · Zijie Geng (University of Science and Technology of China) · Defu Lian (University of Science and Technology of China) · Ziqi Liu (Carnegie Mellon University) · Feng Ju (University of Science and Technology of China) · Yiming Li (Zhejiang University) · Xing Li (Xi'an Jiaotong University) · Enhong Chen (University of Science and Technology of China)
anchor directioncache accuracyinference latencykey-value cache evictionkv cache budgetlongbench benchmarkmemory optimizationperformance degradationprojection techniquesprojection-based scoringspatial relationshipsstate-of-the-art methodstoken importancetoken selectionvalue vectors

Key-Value (KV) cache eviction