Evaluating RAG System Performance

You've built a Retrieval Augmented Generation (RAG) system. But how do you know if it's actually good? This lesson teaches you how to measure its effectiveness.

• RAG systems combine information retrieval with large language models (LLMs).
• Evaluation helps you understand strengths, weaknesses, and areas for improvement.
• It's crucial for building reliable and accurate AI applications.

Evaluating a RAG system means looking at two main components: the retrieval part and the generation part.

• Retrieval Quality: Is the system finding the most relevant information (context) for the user's query?
• Generation Quality: Is the LLM producing accurate, relevant, and coherent answers based on the retrieved context?
• Ultimately, we want to measure the overall user experience and answer quality.

The first step in RAG is getting good context. We use specific metrics to assess how well our system retrieves information.

• Context Relevance: How pertinent is the retrieved information to the user's original query?
• Context Recall: Did the system retrieve *all* the necessary information to answer the question?
• These metrics ensure the LLM has the best possible foundation for generating an answer.

Context Relevance measures if the retrieved documents or snippets are truly related to the user's question.

Imagine asking about 'solar panels' and getting an article about 'wind turbines'. That's low context relevance. High relevance means the retrieved text directly addresses the query's topic.

This is often assessed by comparing the query to each retrieved piece of context.

Context Recall focuses on completeness. It asks: 'Did the RAG system retrieve *all* the critical pieces of information needed to fully answer the user's question?'

Even if retrieved documents are relevant, if they miss a key fact, recall is low. For example, if a question needs 3 facts to be answered completely, and only 2 are retrieved, recall is not perfect.

This metric is especially important for complex questions.

Once the context is retrieved, the LLM generates an answer. We need to evaluate the quality of this generated text.

• Answer Faithfulness (Groundedness): Is the answer purely based on the provided context, or does it 'hallucinate' information?
• Answer Relevance: Is the answer directly addressing the user's original question?
• Answer Coherence: Is the answer well-structured, readable, and grammatically correct?

Faithfulness is critical for RAG. It measures whether every statement in the generated answer can be directly supported by the retrieved context.

If the LLM adds information not found in the context, it's considered unfaithful or 'hallucinated'. This can lead to incorrect or misleading answers.

Example: If the context says 'A is B' and the answer says 'A is C', it's unfaithful.

Answer Relevance ensures the generated answer directly addresses the user's question, without going off-topic.

Answer Coherence assesses the answer's readability, logical flow, and grammatical correctness. A coherent answer is easy to understand and well-organized.

• Relevance: Does it answer the question?
• Coherence: Is it well-written and easy to read?

How do we actually measure these metrics?

• Human Evaluation: Gold standard but slow and expensive. Human annotators manually score answers based on guidelines.
• Automated Evaluation: Faster and scalable. Uses other LLMs or statistical methods to score answers. Can be less nuanced but good for large datasets and frequent checks.
• Often, a combination is used: human evaluation for critical cases, automated for development and large-scale testing.

You're evaluating a RAG system. The user asks: 'What is the capital of France?'

The system retrieves an article about French history that mentions Paris but also includes unrelated facts about Joan of Arc.

The generated answer is: 'Paris is a beautiful city in France.'

Which statement is TRUE about this RAG system's performance?

Great job! You've learned how to evaluate your RAG systems.

• We evaluate both retrieval quality (context relevance, context recall) and generation quality (faithfulness, relevance, coherence).
• Context Relevance ensures retrieved info is on-topic.
• Context Recall checks if all necessary info is retrieved.
• Answer Faithfulness prevents hallucinations by ensuring the answer is grounded in context.
• Answer Relevance keeps the answer focused on the question, and Coherence ensures readability.
• Both human and automated methods are used for evaluation.