Evaluating RAG pipelines with Ragas + LangSmith

Abstraction: Reference-free RAG evaluation using Ragas and LangSmith

Key points:

• Ragas evaluates QA/RAG pipelines (retriever + generator) with 4 LLM-based metrics: context_relevancy and context_recall (retrieval), faithfulness (hallucination) and answer_relevancy (generation); their harmonic mean is the "ragas score."
• Mostly reference-free (only context_recall needs a ground-truth answer), avoiding costly human-annotated golden test sets; addresses biases of LLM judges (preference for own/longer outputs, position bias) noted in "LLMs are not Fair Evaluators."
• Traditional metrics ROUGE/BLEU correlate poorly with human judgment; reference-free LLM eval (e.g. G-Eval) correlates better with less annotation.
• How metrics work under the hood: faithfulness extracts statements then checks each against context; answer_relevancy generates probable questions and measures similarity; context_relevancy is a signal-to-noise sentence ratio; context_recall checks ground-truth statements against retrieved context.
• RagasEvaluatorChain wraps metrics as LangChain evaluators; LangSmith stores datasets, visualizes/traces results, and enables continuous eval by adding production examples. Example on NYC Wikipedia QA scored faithfulness 1.0, answer_relevancy 0.92, context_relevancy 0.07.

Connections: Ragas · Langsmith · Langchain · Retrieval Augmented Generation · LLM Evaluation · LLM As Judge

Source: https://blog.langchain.dev/evaluating-rag-pipelines-with-ragas-langsmith/