Tail-Optimized Caching for LLM Inference
caching policyconversation dynamicsconversation lengthshigh-latency conversationskv cache capacityleast recently usedllm inferencep90 tail latencyp95 tail latencypractical optimizationprompt cachingslo violationsstochastic modeltail latencytail-optimized lrutheoretical justification
Prompt caching is critical for reducing latency and cost in LLM inference