Generation Metrics: Faithfulness and Answer Relevance
Use RAGAS to measure whether generated answers are faithful to the retrieved context and whether they actually address the user's question without hallucinating.
Generation Metrics: Faithfulness and Answer Relevance is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Measuring the Generation Stage
Even when your retriever finds the perfect chunks, the generation stage can still fail. The LLM might ignore the retrieved context and answer from its parametric memory, misinterpret what the context says, or answer a slightly different question than what was asked. Generation metrics quantify these failures independently from retrieval so you can pinpoint and fix each problem. The two primary generation metrics are faithfulness and answer relevance.
Faithfulness: Definition
Faithfulness measures whether every claim in the generated answer can be directly traced back to the retrieved context. A faithful answer introduces no information that is not present in the context. Faithfulness is measured at the claim level: the answer is decomposed into individual atomic statements, and each is checked against the context for support. The faithfulness score is the fraction of claims that are supported.
# Faithfulness = supported_claims / total_claims
example_answer = (
'Employees receive 15 vacation days per year. '
'Remote work is allowed on Wednesdays and Fridays. '
'The CEO is John Smith.'
)
example_context = (
'15 vacation days are granted annually. '
'Remote work is permitted on Wednesdays and Fridays.'
)
# Claim 1: 15 vacation days — SUPPORTED
# Claim 2: Remote work Wed+Fri — SUPPORTED
# Claim 3: CEO is John Smith — NOT IN CONTEXT (hallucinated)
# Faithfulness = 2/3 = 0.67Implementing Faithfulness with LLM-as-Judge
The most practical way to measure faithfulness at scale is to use a powerful LLM as a judge. Prompt the judge LLM to decompose the answer into individual claims, then check each claim against the context. The judge returns a list of claims labeled SUPPORTED or UNSUPPORTED, and you compute the fraction that are supported. This approach scales to thousands of evaluations per hour at a cost of a few cents per evaluation.
import json
def evaluate_faithfulness(answer, context, llm_client):
prompt = (
'Task: Evaluate the faithfulness of an answer against a context.\n\n'
f'Context:\n{context}\n\n'
f'Answer:\n{answer}\n\n'
'Steps:\n'
'1. Break the answer into individual atomic claims.\n'
'2. For each claim, check if it is supported by the context.\n'
'3. Return JSON: {"claims": [{"text": "...", "supported": true/false}], "score": 0.0-1.0}'
)
response = llm_client.chat.completions.create(
model='gpt-4o',
response_format={'type': 'json_object'},
messages=[{'role': 'user', 'content': prompt}]
)
return json.loads(response.choices[0].message.content)Answer Relevance: Definition
Answer relevance measures whether the generated answer actually addresses the user's question. A highly faithful answer might still miss the point — for example, if the user asks 'How do I reset my password?' and the answer explains the company's general security policy in detail without mentioning the password reset procedure. Answer relevance is independent of faithfulness: you can be faithful (only saying things in the context) but irrelevant (not addressing what was asked).
Measuring Answer Relevance
RAGAS measures answer relevance using a clever reverse-generation technique: the judge LLM generates several hypothetical questions that would be answered by the generated response, then measures how similar these generated questions are to the original question using embedding cosine similarity. High similarity between the generated hypothetical questions and the original question indicates high answer relevance.
def evaluate_answer_relevance(question, answer, llm_client, embed_fn):
# Generate hypothetical questions for this answer
prompt = (
f'Given this answer: "{answer}"\n\n'
'Generate 3 questions that this answer would be a good response to.\n'
'Return as JSON: {"questions": ["...", "...", "..."]}'
)
response = llm_client.chat.completions.create(
model='gpt-4o-mini',
response_format={'type': 'json_object'},
messages=[{'role': 'user', 'content': prompt}]
)
hyp_questions = json.loads(response.choices[0].message.content)['questions']
# Measure similarity to original question
orig_embedding = embed_fn(question)
hyp_embeddings = [embed_fn(q) for q in hyp_questions]
similarities = [cosine_similarity(orig_embedding, e) for e in hyp_embeddings]
return sum(similarities) / len(similarities)Context Recall: Did We Retrieve Enough?
Context recall measures whether the retrieved context contains sufficient information to answer the question correctly. It checks the ground truth answer sentence by sentence: can each sentence be attributed to one of the retrieved chunks? High context recall means the retriever found everything needed. Low context recall means key information was missing from the retrieved chunks, so the LLM cannot possibly answer correctly even with perfect generation.
def evaluate_context_recall(ground_truth_answer, retrieved_contexts, llm_client):
prompt = (
f'Ground truth answer:\n{ground_truth_answer}\n\n'
f'Retrieved context:\n{",".join(retrieved_contexts)}\n\n'
'For each sentence in the ground truth answer, determine if it '
'can be attributed to the retrieved context.\n'
'Return JSON: {"sentences": [{"text": "...", "in_context": true/false}], "recall": 0.0-1.0}'
)
response = llm_client.chat.completions.create(
model='gpt-4o',
response_format={'type': 'json_object'},
messages=[{'role': 'user', 'content': prompt}]
)
return json.loads(response.choices[0].message.content)Context Precision: Are Retrieved Chunks Relevant?
Context precision measures whether the retrieved chunks are actually useful for answering the question. High context precision means the retrieved chunks are tightly relevant. Low context precision means many retrieved chunks are off-topic noise that the LLM must read and discard, increasing the risk of confusion. Context precision is measured by checking which retrieved chunks were actually used or referenced in the generated answer.
def evaluate_context_precision(question, answer, retrieved_contexts, llm_client):
precision_scores = []
for i, context in enumerate(retrieved_contexts, 1):
prompt = (
f'Question: {question}\n\n'
f'Context chunk {i}: {context}\n\n'
f'Answer: {answer}\n\n'
'Was this context chunk useful in generating the answer? '
'Return JSON: {"useful": true/false, "reason": "..."}'
)
response = llm_client.chat.completions.create(
model='gpt-4o-mini',
response_format={'type': 'json_object'},
messages=[{'role': 'user', 'content': prompt}]
)
result = json.loads(response.choices[0].message.content)
precision_scores.append(1.0 if result['useful'] else 0.0)
return sum(precision_scores) / len(precision_scores)Using RAGAS for All Metrics at Once
The RAGAS library implements all four core RAG metrics — context precision, context recall, faithfulness, and answer relevance — in a single framework. It handles the LLM judge calls internally. Pass a dataset with questions, generated answers, retrieved contexts, and ground truth answers, and receive a comprehensive score report. RAGAS also supports async evaluation so you can evaluate hundreds of examples in parallel.
from ragas import evaluate
from ragas.metrics import (
faithfulness, answer_relevancy,
context_precision, context_recall
)
from datasets import Dataset
# Build evaluation dataset
eval_samples = [
{
'question': item['question'],
'answer': item['generated_answer'],
'contexts': item['retrieved_texts'],
'ground_truth': item['expected_answer']
}
for item in test_results
]
eval_dataset = Dataset.from_list(eval_samples)
results = evaluate(eval_dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
results.to_pandas().to_csv('eval_results.csv', index=False)Diagnosing with Metric Combinations
The combination of metrics reveals specific system problems. Low faithfulness + high context precision → LLM is ignoring the context and hallucinating (fix the prompt). Low context recall + high faithfulness → retriever is missing key chunks but LLM faithfully reports what it found (fix chunking or embedding). Low answer relevance + high faithfulness → retrieved chunks are tangentially related and the LLM faithfully discusses them but misses the question (improve retrieval filtering or query rewriting).
# Diagnostic matrix
diagnostics = [
{
'condition': 'Low faithfulness + High context precision',
'cause': 'LLM ignoring context, answering from parametric memory',
'fix': 'Strengthen system prompt: "Answer ONLY from the provided context"'
},
{
'condition': 'Low context recall + High faithfulness',
'cause': 'Retriever missing relevant chunks',
'fix': 'Improve chunking strategy, embedding model, or increase top-k'
},
{
'condition': 'Low answer relevance + High context recall',
'cause': 'Retrieved context is off-topic; LLM is answering wrong question',
'fix': 'Add query rewriting or improve metadata filters'
}
]Human Evaluation for Calibration
LLM-as-judge metrics are correlated with human judgment but not perfectly aligned. Calibrate your automated metrics by having 3 human raters score a random sample of 50 outputs on faithfulness and answer relevance using a 1-5 scale. Compute agreement between the LLM judge and the average human rating. If the LLM systematically over- or under-scores relative to humans, apply a correction factor. Calibrated automated metrics give you confidence that score improvements reflect real quality improvements.
from scipy.stats import spearmanr
def calibrate_judge_vs_humans(samples):
'''samples: [{"llm_score": 0.9, "human_score": 4.2}, ...]'''
llm_scores = [s['llm_score'] for s in samples]
human_scores = [s['human_score'] / 5.0 for s in samples] # normalize to 0-1
correlation, pvalue = spearmanr(llm_scores, human_scores)
print(f'LLM-Human Spearman correlation: {correlation:.3f} (p={pvalue:.4f})')
mean_llm = sum(llm_scores) / len(llm_scores)
mean_human = sum(human_scores) / len(human_scores)
bias = mean_llm - mean_human
print(f'LLM bias vs humans: {bias:+.3f}')
return correlation, biasSetting Metric Targets for Production
Before deploying a RAG system to production, define minimum metric thresholds that the system must meet. Common production targets are: faithfulness > 0.90 (fewer than 10% of answer claims are hallucinated), answer relevance > 0.80 (80%+ of answers actually address the question), context precision > 0.60 (most retrieved chunks are relevant), and context recall > 0.75 (most key facts are available in retrieved context). Systems that fail these thresholds should not be deployed without further improvement.
PRODUCTION_THRESHOLDS = {
'faithfulness': 0.90,
'answer_relevancy': 0.80,
'context_precision': 0.60,
'context_recall': 0.75
}
def check_production_readiness(metrics):
ready = True
print('Production readiness check:')
for metric, threshold in PRODUCTION_THRESHOLDS.items():
score = metrics.get(metric, 0)
status = 'PASS' if score >= threshold else 'FAIL'
print(f' {metric}: {score:.2f} (>= {threshold}) -> {status}')
if status == 'FAIL':
ready = False
print(f'Overall: {"READY" if ready else "NOT READY"}')
return readyQuick Check
Test your understanding of AI Engineering concepts from this lesson.
Lesson Recap
In this lesson you learned: faithfulness as the claim-level metric checking whether every answer statement is supported by the retrieved context, answer relevance as the metric measuring whether the answer addresses the actual question, context recall and precision for measuring retrieval quality from the generation perspective, and the RAGAS library for automated multi-metric evaluation. Next up we assemble all these metrics into an automated evaluation harness you can run on every change.
Frequently asked questions
Is the “Generation Metrics: Faithfulness and Answer Relevance” lesson free?
Yes — the full text of “Generation Metrics: Faithfulness and Answer Relevance” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Generation Metrics: Faithfulness and Answer Relevance”?
Use RAGAS to measure whether generated answers are faithful to the retrieved context and whether they actually address the user's question without hallucinating. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Generation Metrics: Faithfulness and Answer Relevance” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Evaluation Matters in RAG
- Retrieval Metrics: Hit Rate, MRR, and NDCG
- Generation Metrics: Faithfulness and Answer Relevance
- Building an Automated Evaluation Harness