Does Prompt Caching Make RAG Obsolete?

Abstraction: Prompt caching economics vs RAG for LLM context management

Key points:

• Anthropic prompt caching: +25% cost on first (cache-write) call, -90% cost on subsequent cache-hit calls; benefits workloads with large static prompts (chat-with-document, long conversations, ML, coding)
• Dynamic prompts using string interpolation invalidate the cache on every call and pay the +25% surcharge — no benefit
• Replacing a vector store with a full-context prompt costs ~$0.10/MB vs ~$0.01/GB for a real database — roughly 10,000x more expensive; not viable at scale today
• Knowledge graphs still preferable over huge contexts for security, structure, and reliable safe LLM application behavior — large context does not resolve those concerns
• Prompt caching makes LLM programming more complex overall, not simpler

Connections: Anthropic · Prompt Caching · Retrieval Augmented Generation · Large Language Models · Knowledge Graphs

Source: https://timkellogg.me/blog/2024/08/14/prompt-caching