Llamar.ai: A deep dive into the (in)feasibility of RAG with LLMs

Abstraction: RAG product prototype built and shut down due to GPT-4 API cost infeasibility

Key points:

• Built a working RAG prototype (Llamar.ai) over 4 months: a ChatGPT-like agent with deep domain knowledge using in-context learning across multiple corpora (docs, forums, blogs, web search)
• Core RAG challenge: context window limits; GPT-3.5 had 4k tokens (inadequate), 16k version costs 4x per-query; GPT-4 8k window had noticeably superior quality but 20x the cost of GPT-3.5
• At hundreds of thousands of monthly queries, cost difference is $5k/month (GPT-3.5) vs. $100k/month (GPT-4) — one model feasible but inadequate, the other adequate but infeasible
• "Lost in the Middle" attention problem: long-context LLMs over-emphasize start/end of context, ignore the middle; addressed with smarter prompting
• LangChain useful for prototypes but too complex for production; replaced with custom implementation; also used LlamaIndex and ReAct-style autonomous plugin selection
• Conclusion: RAG use case on enterprise scale was infeasible in 2023; would require ~95% cost reduction in GPT-4 API pricing to become viable

Connections: Openai · Langchain · Llamaindex · Retrieval Augmented Generation · Large Language Models · In Context Learning

Source: https://www.bolddata.org/blog/llamar-ai-infeasibility-retrieval-augmented-generation-llm/