Why Evaluation Matters in RAG

A RAG system can feel like it is working because it returns fluent, plausible-sounding answers. But without measurement, you have no idea whether it is actually retrieving the right chunks or generating faithful answers. Teams that skip evaluation often spend months tweaking chunking strategies and prompt formats based on gut feeling, only to discover they made things worse. Rigorous evaluation is what turns RAG development from guessing into engineering.

RAG has two distinct stages that can fail independently: retrieval and generation. Retrieval fails when the relevant chunks are not ranked in the top-K results — the LLM cannot generate a good answer if the right information was never retrieved. Generation fails when the correct chunks were retrieved but the LLM ignored them, misread them, or added hallucinated information. You need separate metrics for each stage to pinpoint which component is causing the problem.

Measuring only the final answer quality hides where failures come from. Suppose your system gives wrong answers 30% of the time. Is that because retrieval is missing the right chunks, or because the LLM is ignoring good chunks? If you only know the final error rate, you cannot know which component to fix. Instrument both stages separately: measure retrieval quality with golden datasets and generation quality with faithfulness scores.

Evaluation requires a golden dataset: a set of question-answer pairs where you know the correct answer and, ideally, which document and chunk the answer comes from. For a minimum viable evaluation set, collect 50-100 questions representative of real user queries. Answer them by hand or by reading the source documents. Include diverse question types: factual lookups, comparisons, multi-hop reasoning, and out-of-domain questions the system should refuse to answer.

Creating 100 questions manually is tedious. Accelerate this with a data generation LLM: feed each document chunk to GPT-4o and ask it to generate 3-5 diverse questions whose answers can be found in that chunk, plus the expected answer text. Review a sample manually to catch quality issues. This approach scales to thousands of questions quickly, though it may miss edge cases that only real users would ask.

Hit rate@K is the fraction of questions for which at least one relevant chunk appears in the top-K retrieval results. It is the simplest and most intuitive retrieval metric. A hit rate@5 of 85% means that for 85 out of 100 questions, the relevant chunk was among the top 5 results. Track hit rate separately for different document types, query lengths, and topic categories to find where your retriever struggles most.

Faithfulness measures whether the generated answer contains only information that can be verified in the retrieved context. An unfaithful answer adds facts the context does not support — that is hallucination. Score faithfulness by having a judge LLM (or human evaluator) check each sentence in the answer against the context and flag any claims not supported by the retrieved chunks. A faithfulness score of 95%+ is the target for production systems.

Answer relevance measures whether the generated answer actually addresses the user's question. A highly faithful answer might still miss the point by answering a related but different question. Measure relevance separately from faithfulness. Use a judge LLM to rate whether the answer directly addresses what was asked, using a 1-5 scale. Low answer relevance often indicates a problem with the prompt structure or the retrieved context not actually containing the answer.

Evaluation is most valuable when you track metrics over time as you make changes. Store evaluation results in a database or spreadsheet with timestamps and version labels (e.g., chunk_size=500, embed=3-small, k=5). When you try a new chunking strategy or embedding model, run the same evaluation and compare. This prevents regression — you might improve faithfulness but accidentally reduce hit rate. Always run the full evaluation before merging changes to production.

Beyond specific metrics, evaluation requires a mindset shift: treat RAG as a machine learning system with measurable performance, not a chatbot you subjectively assess by chatting with it. Define success criteria upfront (e.g., hit rate@5 > 85%, faithfulness > 95%). Establish a test set that stays frozen and is never used for development decisions. Reserve a separate dev set for iteration. This discipline separates teams that ship reliable RAG from teams that ship impressive demos that fail in production.

A critical discipline in machine learning that also applies to RAG evaluation is the train-dev-test split. Your test set should be completely frozen — never used to make development decisions. Use a separate dev set for experimenting with chunking strategies, prompt changes, and embedding models. Only run the test set when you believe a change is production-ready. This separation prevents overfitting your RAG pipeline to the test set and ensures your final reported metrics reflect genuine generalization.

Test your understanding of AI Engineering concepts from this lesson.

In this lesson you learned: the two independent failure modes of retrieval and generation that require separate metrics, how to build a golden dataset of question-answer-chunk triples for objective evaluation, hit rate as the core retrieval metric and faithfulness and answer relevance as the core generation metrics, and the importance of tracking metrics over time to prevent regressions. Next up we implement specific retrieval metrics including MRR and NDCG.