Retrieval Metrics: Hit Rate, MRR, and NDCG
Build a golden dataset of queries and relevant documents, then compute hit rate, mean reciprocal rank, and NDCG to measure how often your retriever finds the right chunks.
Retrieval Metrics: Hit Rate, MRR, and NDCG is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Different Retrieval Metrics?
Hit rate tells you whether a relevant chunk appeared anywhere in the top-K results, but it does not tell you where in the ranking it appeared. A system that always puts the best chunk at rank 5 is worse than one that consistently puts it at rank 1, even if both have the same hit rate. More nuanced metrics like MRR and NDCG capture ranking quality, rewarding systems that put the most relevant chunks at the top where the LLM and users are most likely to use them.
Hit Rate @K: Review and Implementation
Hit rate@K is the simplest metric: for what fraction of queries does at least one relevant chunk appear in the top-K results? It gives a binary signal per query and is easy to interpret. Compute it by checking set intersection between retrieved IDs and the known relevant IDs. Use K=5 as the default since most RAG systems retrieve 5 chunks. Compare hit rate@1, @3, and @5 to understand how sensitivity changes as you expand the retrieval window.
def hit_rate_at_k(golden_dataset, retriever, k=5):
hits = 0
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=k)
retrieved_ids = [r['id'] for r in retrieved[:k]]
relevant_ids = set(item['relevant_chunk_ids'])
if any(rid in relevant_ids for rid in retrieved_ids):
hits += 1
return hits / len(golden_dataset)
for k in [1, 3, 5, 10]:
hr = hit_rate_at_k(golden_dataset, retriever, k=k)
print(f'Hit rate@{k}: {hr:.1%}')Mean Reciprocal Rank (MRR)
MRR (Mean Reciprocal Rank) measures the average reciprocal of the rank at which the first relevant chunk appears. If the relevant chunk is at rank 1, the reciprocal rank is 1/1 = 1.0. At rank 2 it is 0.5, at rank 5 it is 0.2. MRR is averaged over all queries. Higher MRR means the retriever consistently puts relevant chunks near the top, which is important because the LLM pays more attention to context positioned early in the prompt.
def mean_reciprocal_rank(golden_dataset, retriever, top_k=10):
reciprocal_ranks = []
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=top_k)
retrieved_ids = [r['id'] for r in retrieved]
relevant_ids = set(item['relevant_chunk_ids'])
rr = 0.0
for rank, rid in enumerate(retrieved_ids, start=1):
if rid in relevant_ids:
rr = 1.0 / rank
break # only the first relevant result counts
reciprocal_ranks.append(rr)
mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)
print(f'MRR@{top_k}: {mrr:.3f}')
return mrrInterpreting MRR Scores
MRR scores have intuitive interpretations: MRR = 1.0 means the first relevant chunk is always at rank 1 (perfect). MRR = 0.5 means it is typically at rank 2. MRR = 0.25 means it is typically at rank 4 — the relevant information appears far enough down that it may be truncated from the context. For RAG, target MRR > 0.7 to ensure that your most relevant chunk is consistently within the first two positions.
# MRR interpretation table
mrr_interpretations = {
1.0: 'Perfect — relevant chunk always at rank 1',
0.5: 'Good — typically at rank 2',
0.33: 'Acceptable — typically at rank 3',
0.25: 'Weak — typically at rank 4',
0.1: 'Poor — relevant chunk rarely near the top'
}
for score, description in mrr_interpretations.items():
print(f'MRR {score:.2f}: {description}')Precision @K
Precision@K measures what fraction of the K retrieved chunks are actually relevant. Unlike hit rate (which is binary), precision@K measures the signal-to-noise ratio in your retrieval results. Low precision means the LLM receives irrelevant context alongside relevant chunks, increasing the risk of confusion or prompt injection. For RAG, precision@5 > 0.6 is a healthy target.
def precision_at_k(golden_dataset, retriever, k=5):
precisions = []
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=k)
retrieved_ids = [r['id'] for r in retrieved[:k]]
relevant_ids = set(item['relevant_chunk_ids'])
relevant_retrieved = sum(
1 for rid in retrieved_ids if rid in relevant_ids
)
precision = relevant_retrieved / k
precisions.append(precision)
mean_precision = sum(precisions) / len(precisions)
print(f'Precision@{k}: {mean_precision:.3f}')
return mean_precisionDiscounted Cumulative Gain (DCG)
DCG is a ranked list evaluation metric that rewards putting more relevant chunks higher up. It sums the relevance scores of retrieved chunks, but discounts them logarithmically by position: rank 1 gets full credit, rank 2 gets log(2) discount, and so on. Higher-relevance chunks at the top yield higher DCG. The normalized version (NDCG) divides by the ideal DCG (the best possible ranking) to produce a score between 0 and 1.
import math
def dcg_at_k(relevances, k):
'''relevances[i] = 1 if chunk at rank i+1 is relevant, else 0'''
dcg = 0.0
for i, rel in enumerate(relevances[:k]):
# rank is i+1, discount is log2(rank + 1)
dcg += rel / math.log2(i + 2)
return dcg
def ndcg_at_k(retrieved_ids, relevant_ids, k):
relevances = [1 if rid in relevant_ids else 0
for rid in retrieved_ids[:k]]
actual_dcg = dcg_at_k(relevances, k)
ideal_dcg = dcg_at_k([1] * min(len(relevant_ids), k), k)
return actual_dcg / ideal_dcg if ideal_dcg > 0 else 0.0Computing NDCG Across the Dataset
NDCG@K is the gold standard retrieval metric for search systems. It captures both relevance and ranking position simultaneously. An NDCG@5 of 0.85 means your retriever is performing at 85% of the theoretical best ranking on average. NDCG handles cases where there are multiple relevant chunks per query (each gets a relevance score) and penalizes systems that find the relevant chunks but rank them too low.
def mean_ndcg_at_k(golden_dataset, retriever, k=5):
ndcg_scores = []
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=k)
retrieved_ids = [r['id'] for r in retrieved[:k]]
relevant_ids = set(item['relevant_chunk_ids'])
score = ndcg_at_k(retrieved_ids, relevant_ids, k)
ndcg_scores.append(score)
mean = sum(ndcg_scores) / len(ndcg_scores)
print(f'NDCG@{k}: {mean:.4f}')
return mean
# Compute all retrieval metrics together
hit_rate = hit_rate_at_k(golden_dataset, retriever, k=5)
mrr = mean_reciprocal_rank(golden_dataset, retriever, top_k=5)
ndcg = mean_ndcg_at_k(golden_dataset, retriever, k=5)Using RAGAS for Automated Metrics
The RAGAS library provides a production-ready evaluation framework for RAG systems. It implements context precision, context recall, faithfulness, and answer relevance metrics using an LLM-as-judge approach. Pass your questions, answers, retrieved contexts, and ground truth answers to RAGAS and receive a full evaluation report with scores for every metric. This is the fastest way to set up a comprehensive RAG evaluation pipeline.
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy
)
from datasets import Dataset
eval_data = Dataset.from_list([
{
'question': item['question'],
'answer': item['generated_answer'],
'contexts': item['retrieved_texts'],
'ground_truth': item['expected_answer']
}
for item in golden_dataset_with_answers
])
results = evaluate(
eval_data,
metrics=[context_precision, context_recall, faithfulness, answer_relevancy]
)
print(results)Slicing Metrics by Query Category
Overall average metrics hide important patterns. Segment your golden dataset into categories — factual lookups, comparisons, procedural how-to questions, and out-of-scope questions — and compute metrics separately for each. You may find that hit rate is 95% for factual lookups but only 60% for multi-hop comparisons. Category-level metrics reveal the specific failure modes that aggregate scores conceal.
from collections import defaultdict
def evaluate_by_category(golden_dataset, retriever):
by_category = defaultdict(list)
for item in golden_dataset:
category = item.get('category', 'unknown')
retrieved = retriever.retrieve(item['question'], top_k=5)
retrieved_ids = {r['id'] for r in retrieved}
hit = bool(retrieved_ids & set(item['relevant_chunk_ids']))
by_category[category].append(hit)
print('Hit rate by category:')
for cat, hits in sorted(by_category.items()):
hr = sum(hits) / len(hits)
print(f' {cat}: {hr:.1%} ({sum(hits)}/{len(hits)})')When Metrics Disagree
Sometimes metrics send conflicting signals. You might improve NDCG (better ranking) while hit rate stays flat (same number of misses). This happens when optimization moves relevant chunks from rank 6 to rank 2 without bringing previously missed queries into the top-5. In such cases, check if your optimization helped on the queries that were already succeeding and neglected the failing ones. Always examine individual failure examples alongside aggregate metrics.
def analyze_failures(golden_dataset, retriever, top_k=5):
failures = []
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=top_k)
retrieved_ids = {r['id'] for r in retrieved}
relevant_ids = set(item['relevant_chunk_ids'])
if not (retrieved_ids & relevant_ids):
failures.append({
'question': item['question'],
'expected_chunks': list(relevant_ids),
'retrieved_chunks': [r['id'] for r in retrieved],
'top_score': retrieved[0]['score'] if retrieved else None
})
print(f'Failures: {len(failures)}/{len(golden_dataset)}')
return failuresRecall @K: Completeness of Retrieval
Recall@K measures what fraction of all relevant chunks were retrieved in the top-K results. If a question has 3 relevant chunks in the index and your retriever returns 2 of them in the top-5, recall@5 is 2/3 = 0.67. High recall matters when the LLM needs multiple pieces of evidence to synthesize a complete answer — missing even one key chunk can make the answer incomplete. Balance precision and recall by tuning K: larger K improves recall but reduces precision.
def recall_at_k(golden_dataset, retriever, k=5):
recalls = []
for item in golden_dataset:
retrieved = retriever.retrieve(item['question'], top_k=k)
retrieved_ids = set(r['id'] for r in retrieved[:k])
relevant_ids = set(item['relevant_chunk_ids'])
if not relevant_ids:
continue # skip items with no annotated relevant chunks
retrieved_relevant = retrieved_ids & relevant_ids
recall = len(retrieved_relevant) / len(relevant_ids)
recalls.append(recall)
mean_recall = sum(recalls) / len(recalls)
print(f'Recall@{k}: {mean_recall:.3f}')
return mean_recallQuick Check
Test your understanding of AI Engineering concepts from this lesson.
Lesson Recap
In this lesson you learned: hit rate@K as the binary presence metric, MRR for measuring average first-relevant-rank position, precision@K for retrieval signal-to-noise ratio, NDCG@K as the gold-standard ranked metric, and the RAGAS library for automating RAG evaluation with context precision, recall, faithfulness, and answer relevance. Next up we dive deeper into generation metrics and faithfulness measurement.
Frequently asked questions
Is the “Retrieval Metrics: Hit Rate, MRR, and NDCG” lesson free?
Yes — the full text of “Retrieval Metrics: Hit Rate, MRR, and NDCG” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Retrieval Metrics: Hit Rate, MRR, and NDCG”?
Build a golden dataset of queries and relevant documents, then compute hit rate, mean reciprocal rank, and NDCG to measure how often your retriever finds the right chunks. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Retrieval Metrics: Hit Rate, MRR, and NDCG” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Evaluation Matters in RAG
- Retrieval Metrics: Hit Rate, MRR, and NDCG
- Generation Metrics: Faithfulness and Answer Relevance
- Building an Automated Evaluation Harness