재순위 지정의 영향 측정
단일 단계 검색과 재순위 지정을 적용한 2단계 검색을 전후 벤치마크로 비교하고, NDCG, MRR, 종단 간 답변 품질을 측정합니다.
재순위 지정의 영향 측정은(는) CoddyKit의 무료 AI Engineering Academy 강의입니다. 이것은 4개 중 4번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 AI Engineering Academy 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. AI Engineering Academy 강의에는 총 4개의 강의가 포함되어 있습니다.
이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.
Why Measure Re-ranking Impact?
Re-ranking adds latency and cost to your pipeline. Without measurement, you cannot answer whether the added complexity is worth it. Benchmarking quantifies the improvement in retrieval quality and end-to-end answer quality so you can make an informed decision. It also reveals which query types benefit most, enabling you to apply re-ranking selectively rather than on every request.
Building a Golden Test Set
A reliable benchmark requires a golden test set: a collection of queries paired with the document IDs that are known to be relevant. Create it by sampling real user queries from your application logs, identifying the relevant documents manually or with expert annotation, and organizing them into a structured format. A test set of 50-200 queries is sufficient for most RAG evaluation purposes.
golden_test_set = [
{
'query': 'How does pgvector HNSW indexing improve search speed?',
'relevant_doc_ids': ['doc_042', 'doc_107'],
},
{
'query': 'What is the difference between BM25 and dense retrieval?',
'relevant_doc_ids': ['doc_015'],
},
{
'query': 'How to implement reciprocal rank fusion in Python?',
'relevant_doc_ids': ['doc_093', 'doc_094'],
},
# ... 47 more entries
]
print(f'Test set size: {len(golden_test_set)} queries')
print(f'Avg relevant docs per query: {sum(len(e["relevant_doc_ids"]) for e in golden_test_set) / len(golden_test_set):.1f}')Retrieval Metrics: NDCG, MRR, Hit Rate
Use three complementary metrics to evaluate retrieval quality. Hit Rate at K measures whether at least one relevant document appears in the top K results. MRR (Mean Reciprocal Rank) measures the average reciprocal of the rank of the first relevant document. NDCG at K (Normalized Discounted Cumulative Gain) measures ranking quality with higher positions weighted more than lower ones.
def compute_retrieval_metrics(results: list[str], relevant_ids: set, k: int = 5):
results_at_k = results[:k]
relevant_found = [r for r in results_at_k if r in relevant_ids]
# Hit rate
hit = 1 if relevant_found else 0
# MRR
rr = 0
for i, doc_id in enumerate(results_at_k, start=1):
if doc_id in relevant_ids:
rr = 1.0 / i
break
# NDCG (binary relevance)
import math
dcg = sum(
1.0 / math.log2(i + 1)
for i, doc_id in enumerate(results_at_k, start=1)
if doc_id in relevant_ids
)
ideal = sum(1.0 / math.log2(i + 1) for i in range(1, min(len(relevant_ids), k) + 1))
ndcg = dcg / ideal if ideal > 0 else 0
return {'hit': hit, 'rr': rr, 'ndcg': ndcg}Baseline: Single-Stage Dense Retrieval
Before measuring the impact of re-ranking, establish a baseline using single-stage dense retrieval. Run every query in your test set through the bi-encoder retriever, collect the ranked document IDs, and compute average NDCG, MRR, and hit rate. This baseline tells you how much improvement you are starting from — if your baseline is already 0.95 NDCG@5, re-ranking has little room to improve.
def evaluate_pipeline(retriever_fn, test_set: list[dict], k: int = 5) -> dict:
all_metrics = []
for entry in test_set:
query = entry['query']
relevant = set(entry['relevant_doc_ids'])
results = retriever_fn(query, top_k=k)
result_ids = [r['id'] for r in results]
metrics = compute_retrieval_metrics(result_ids, relevant, k)
all_metrics.append(metrics)
n = len(all_metrics)
return {
f'hit_rate@{k}': sum(m['hit'] for m in all_metrics) / n,
f'mrr@{k}': sum(m['rr'] for m in all_metrics) / n,
f'ndcg@{k}': sum(m['ndcg'] for m in all_metrics) / n,
}Running the Before-and-After Benchmark
Run the same evaluation function against both your single-stage retriever and your two-stage retriever with re-ranking. Print results side by side to make the improvement (or lack thereof) immediately visible. Track latency per query alongside quality metrics — a retrieval quality gain of 5 percent may not be worth a latency increase of 400ms depending on your application's SLA requirements.
import time
def evaluate_with_latency(retriever_fn, test_set, k=5):
metrics_list = []
latencies = []
for entry in test_set:
t0 = time.perf_counter()
results = retriever_fn(entry['query'], top_k=k)
latencies.append((time.perf_counter() - t0) * 1000)
result_ids = [r['id'] for r in results]
metrics_list.append(compute_retrieval_metrics(
result_ids, set(entry['relevant_doc_ids']), k
))
n = len(metrics_list)
return {
f'hit_rate@{k}': sum(m['hit'] for m in metrics_list) / n,
f'ndcg@{k}': sum(m['ndcg'] for m in metrics_list) / n,
'p50_latency_ms': sorted(latencies)[n // 2],
'p99_latency_ms': sorted(latencies)[int(n * 0.99)],
}
baseline = evaluate_with_latency(dense_retrieval_fn, golden_test_set)
two_stage = evaluate_with_latency(two_stage_fn, golden_test_set)
print('Baseline:', baseline)
print('Two-stage:', two_stage)Interpreting NDCG Improvements
Typical improvements from adding cross-encoder re-ranking to dense retrieval range from 0.05 to 0.15 absolute NDCG@5, which translates to roughly 5-15 percentage points. The improvement is larger when: (1) your queries are diverse with many paraphrases, (2) your corpus has many near-relevant chunks, or (3) your first-stage retriever is weak. If you see less than 0.02 NDCG improvement, the benefit may not justify the added complexity.
# Interpreting benchmark results
example_results = {
'baseline': {'hit_rate@5': 0.78, 'ndcg@5': 0.64, 'p99_latency_ms': 35},
'two_stage': {'hit_rate@5': 0.89, 'ndcg@5': 0.77, 'p99_latency_ms': 287},
}
delta_ndcg = example_results['two_stage']['ndcg@5'] - example_results['baseline']['ndcg@5']
delta_latency = example_results['two_stage']['p99_latency_ms'] - example_results['baseline']['p99_latency_ms']
print(f'NDCG improvement: +{delta_ndcg:.2f} (+{delta_ndcg/example_results["baseline"]["ndcg@5"]*100:.0f}%)')
print(f'Latency increase: +{delta_latency}ms')
# NDCG improvement: +0.13 (+20%) — clearly worth the 252ms latency costEnd-to-End Answer Quality Measurement
Retrieval metrics measure whether the right documents were retrieved, but the ultimate measure is end-to-end answer quality. Use an LLM judge to evaluate whether answers generated from re-ranked context are more correct and faithful than answers from single-stage context. Score on a 1-5 scale for correctness, faithfulness, and relevance, then average across your test set.
from openai import OpenAI
client = OpenAI()
JUDGE_PROMPT = '''
Rate the following answer on a scale of 1-5 for correctness and faithfulness to the context.
Question: {question}
Context: {context}
Answer: {answer}
Ground truth: {ground_truth}
Return a JSON with fields: {"correctness": int, "faithfulness": int, "explanation": str}
'''
def judge_answer(question, context, answer, ground_truth):
prompt = JUDGE_PROMPT.format(
question=question, context=context,
answer=answer, ground_truth=ground_truth,
)
response = client.chat.completions.create(
model='gpt-4o',
messages=[{'role': 'user', 'content': prompt}],
response_format={'type': 'json_object'},
)
import json
return json.loads(response.choices[0].message.content)Stratified Analysis by Query Type
Average metrics hide important differences across query types. Segment your test set into categories — factual lookups (who, what, when), procedural queries (how to), conceptual queries (why, explain), and technical queries (error codes, API names) — and compute metrics separately for each group. Re-ranking often helps most on conceptual and procedural queries where semantic understanding matters more than keyword matching.
def stratified_eval(retriever_fn, test_set, k=5):
groups = {'factual': [], 'procedural': [], 'conceptual': [], 'technical': []}
for entry in test_set:
q = entry['query'].lower()
if any(w in q for w in ['how to', 'how do', 'steps to']):
groups['procedural'].append(entry)
elif any(w in q for w in ['why', 'explain', 'what is the reason']):
groups['conceptual'].append(entry)
elif any(c.isupper() for c in q.split()) or 'error' in q:
groups['technical'].append(entry)
else:
groups['factual'].append(entry)
for group_name, group_entries in groups.items():
if group_entries:
metrics = evaluate_pipeline(retriever_fn, group_entries, k)
print(f'{group_name} ({len(group_entries)} queries): ndcg@{k}={metrics[f"ndcg@{k}"]:.3f}')Regression Testing with CI Integration
Run your retrieval benchmark as a regression test in CI. Set minimum acceptable thresholds for NDCG@5, MRR, and hit rate. Any pipeline change that causes metrics to drop below the threshold fails the CI build, preventing retrieval quality regressions from shipping to production. This is especially important after changing chunk sizes, embedding models, or re-ranking models.
# pytest integration for retrieval quality gates
import pytest
MIN_NDCG_5 = 0.70
MIN_HIT_RATE_5 = 0.85
def test_retrieval_quality_meets_threshold():
metrics = evaluate_pipeline(production_retriever_fn, golden_test_set, k=5)
assert metrics['ndcg@5'] >= MIN_NDCG_5, (
f'NDCG@5 {metrics["ndcg@5"]:.3f} below threshold {MIN_NDCG_5}'
)
assert metrics['hit_rate@5'] >= MIN_HIT_RATE_5, (
f'Hit rate {metrics["hit_rate@5"]:.3f} below threshold {MIN_HIT_RATE_5}'
)
# Run with: pytest tests/test_retrieval.py -vVisualizing Retrieval Metrics
Raw numbers are hard to interpret across multiple experiments. Create a simple comparison table or bar chart that shows NDCG, MRR, hit rate, and latency side by side for baseline, hybrid-only, and hybrid-with-reranking pipelines. Tracking these metrics over time as you make improvements creates a retrieval improvement history that guides future optimization decisions.
def print_comparison_table(results: dict[str, dict]):
headers = ['Pipeline', 'NDCG@5', 'MRR@5', 'Hit@5', 'P99 ms']
print('|'.join(f'{h:20}' for h in headers))
print('-' * (len(headers) * 21))
for pipeline_name, metrics in results.items():
row = [
pipeline_name,
f'{metrics.get("ndcg@5", 0):.3f}',
f'{metrics.get("mrr@5", 0):.3f}',
f'{metrics.get("hit_rate@5", 0):.3f}',
f'{metrics.get("p99_latency_ms", 0):.0f}',
]
print('|'.join(f'{v:20}' for v in row))
results = {
'Dense only': {'ndcg@5': 0.64, 'mrr@5': 0.68, 'hit_rate@5': 0.78, 'p99_latency_ms': 35},
'Hybrid RRF': {'ndcg@5': 0.71, 'mrr@5': 0.74, 'hit_rate@5': 0.84, 'p99_latency_ms': 55},
'Hybrid + Rerank': {'ndcg@5': 0.77, 'mrr@5': 0.81, 'hit_rate@5': 0.89, 'p99_latency_ms': 287},
}
print_comparison_table(results)Acting on Benchmark Results
After running benchmarks, use the results to make concrete decisions. If re-ranking improves NDCG by less than 0.03, skip it and focus on improving the first stage. If hit rate is low, your first stage is missing relevant documents — increase the candidate set size or switch to hybrid retrieval. If end-to-end answer quality improves significantly despite modest retrieval gains, the re-ranker may be surfacing highly relevant sentences the LLM uses effectively even if ranked position is unchanged.
Quick Check
Test your understanding of measuring retrieval and re-ranking impact from this lesson.
Lesson Recap
In this lesson you learned: building a golden test set with known relevant documents is essential before measuring retrieval quality, NDCG, MRR, and hit rate are the three core retrieval metrics that together measure ranking quality comprehensively, and end-to-end answer quality using an LLM judge provides the ultimate measure of pipeline improvement. Always run retrieval benchmarks as regression tests in CI. Next up we explore LLM streaming to display tokens as they are generated.
자주 묻는 질문
“재순위 지정의 영향 측정” 강의는 무료인가요?
네 — “재순위 지정의 영향 측정” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 AI Engineering Academy 강의 전체를 잠금 해제할 수 있습니다. AI Engineering Academy 강의에는 총 4개의 강의가 포함되어 있습니다.
“재순위 지정의 영향 측정”에서 뭘 배우나요?
단일 단계 검색과 재순위 지정을 적용한 2단계 검색을 전후 벤치마크로 비교하고, NDCG, MRR, 종단 간 답변 품질을 측정합니다. 브라우저에서 직접 실행하는 실습 코드로 AI Engineering Academy을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.
AI Engineering Academy을(를) 시작하는 데 경험이 필요한가요?
사전 경험은 필요하지 않습니다. CoddyKit의 AI Engineering Academy은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 4번째 강의입니다.
“재순위 지정의 영향 측정” 강의는 얼마나 걸리나요?
대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.
이 AI Engineering Academy 강의에서 코드를 작성하고 실행할 수 있나요?
네. 모든 AI Engineering Academy 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.