Learned Prefix Caching for Efficient LLM Inference

Dongsheng Yang (Princeton University) · Austin Li (Princeton University) · Kai Li (Tsinghua University, Tsinghua University) · Wyatt Lloyd (Princeton University)

cache managementcache sizesconversational content analysiseviction algorithmhit ratiosinference costs large language modellast access timestampslearned methodleast-recently-usedoptimal algorithmpredictive guidanceprefilling throughputprefix cachingreal-world datasets

Prefix caching is a key technique for reducing Large Language Model (LLM) inference costs. However, the prevalent least-recently-used (LRU) eviction algorithm has a large gap to the optimal algorithm. This paper introduces LPC, the first learned method to perform LLM prefix cache eviction. LPC leverages conversational content analysis to provide predictive guidance for eviction, determining which conversations are likely to continue. These insights, combined with last access timestamps, inform more effective cache management. Extensive evaluations across three real-world datasets demonstrate that LPC achieves 18-47% reductions in required cache sizes for equivalent hit ratios and has an 11% improvement in LLM prefilling throughput in an emulated environment.