0Pricing
AI Engineering Academy · Lección

Medición del impacto de la reordenación

Ejecute una evaluación comparativa antes y después, contrastando la recuperación en una sola etapa con la recuperación en dos etapas y reordenación, y mida NDCG, MRR y la calidad de las respuestas de principio a fin.

Medición del impacto de la reordenación es una lección gratuita de AI Engineering Academy en CoddyKit. Esta es la lección 4 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de AI Engineering Academy, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de AI Engineering Academy incluye 4 lecciones en total.

Partes de esta lección aún no han sido traducidas y se muestran en inglés.

Why Measure Re-ranking Impact?

Re-ranking adds latency and cost to your pipeline. Without measurement, you cannot answer whether the added complexity is worth it. Benchmarking quantifies the improvement in retrieval quality and end-to-end answer quality so you can make an informed decision. It also reveals which query types benefit most, enabling you to apply re-ranking selectively rather than on every request.

Building a Golden Test Set

A reliable benchmark requires a golden test set: a collection of queries paired with the document IDs that are known to be relevant. Create it by sampling real user queries from your application logs, identifying the relevant documents manually or with expert annotation, and organizing them into a structured format. A test set of 50-200 queries is sufficient for most RAG evaluation purposes.

golden_test_set = [
    {
        'query': 'How does pgvector HNSW indexing improve search speed?',
        'relevant_doc_ids': ['doc_042', 'doc_107'],
    },
    {
        'query': 'What is the difference between BM25 and dense retrieval?',
        'relevant_doc_ids': ['doc_015'],
    },
    {
        'query': 'How to implement reciprocal rank fusion in Python?',
        'relevant_doc_ids': ['doc_093', 'doc_094'],
    },
    # ... 47 more entries
]

print(f'Test set size: {len(golden_test_set)} queries')
print(f'Avg relevant docs per query: {sum(len(e["relevant_doc_ids"]) for e in golden_test_set) / len(golden_test_set):.1f}')

Retrieval Metrics: NDCG, MRR, Hit Rate

Use three complementary metrics to evaluate retrieval quality. Hit Rate at K measures whether at least one relevant document appears in the top K results. MRR (Mean Reciprocal Rank) measures the average reciprocal of the rank of the first relevant document. NDCG at K (Normalized Discounted Cumulative Gain) measures ranking quality with higher positions weighted more than lower ones.

def compute_retrieval_metrics(results: list[str], relevant_ids: set, k: int = 5):
    results_at_k = results[:k]
    relevant_found = [r for r in results_at_k if r in relevant_ids]

    # Hit rate
    hit = 1 if relevant_found else 0

    # MRR
    rr = 0
    for i, doc_id in enumerate(results_at_k, start=1):
        if doc_id in relevant_ids:
            rr = 1.0 / i
            break

    # NDCG (binary relevance)
    import math
    dcg = sum(
        1.0 / math.log2(i + 1)
        for i, doc_id in enumerate(results_at_k, start=1)
        if doc_id in relevant_ids
    )
    ideal = sum(1.0 / math.log2(i + 1) for i in range(1, min(len(relevant_ids), k) + 1))
    ndcg = dcg / ideal if ideal > 0 else 0

    return {'hit': hit, 'rr': rr, 'ndcg': ndcg}

Baseline: Single-Stage Dense Retrieval

Before measuring the impact of re-ranking, establish a baseline using single-stage dense retrieval. Run every query in your test set through the bi-encoder retriever, collect the ranked document IDs, and compute average NDCG, MRR, and hit rate. This baseline tells you how much improvement you are starting from — if your baseline is already 0.95 NDCG@5, re-ranking has little room to improve.

def evaluate_pipeline(retriever_fn, test_set: list[dict], k: int = 5) -> dict:
    all_metrics = []

    for entry in test_set:
        query = entry['query']
        relevant = set(entry['relevant_doc_ids'])

        results = retriever_fn(query, top_k=k)
        result_ids = [r['id'] for r in results]

        metrics = compute_retrieval_metrics(result_ids, relevant, k)
        all_metrics.append(metrics)

    n = len(all_metrics)
    return {
        f'hit_rate@{k}': sum(m['hit'] for m in all_metrics) / n,
        f'mrr@{k}': sum(m['rr'] for m in all_metrics) / n,
        f'ndcg@{k}': sum(m['ndcg'] for m in all_metrics) / n,
    }

Running the Before-and-After Benchmark

Run the same evaluation function against both your single-stage retriever and your two-stage retriever with re-ranking. Print results side by side to make the improvement (or lack thereof) immediately visible. Track latency per query alongside quality metrics — a retrieval quality gain of 5 percent may not be worth a latency increase of 400ms depending on your application's SLA requirements.

import time

def evaluate_with_latency(retriever_fn, test_set, k=5):
    metrics_list = []
    latencies = []

    for entry in test_set:
        t0 = time.perf_counter()
        results = retriever_fn(entry['query'], top_k=k)
        latencies.append((time.perf_counter() - t0) * 1000)

        result_ids = [r['id'] for r in results]
        metrics_list.append(compute_retrieval_metrics(
            result_ids, set(entry['relevant_doc_ids']), k
        ))

    n = len(metrics_list)
    return {
        f'hit_rate@{k}': sum(m['hit'] for m in metrics_list) / n,
        f'ndcg@{k}': sum(m['ndcg'] for m in metrics_list) / n,
        'p50_latency_ms': sorted(latencies)[n // 2],
        'p99_latency_ms': sorted(latencies)[int(n * 0.99)],
    }

baseline = evaluate_with_latency(dense_retrieval_fn, golden_test_set)
two_stage = evaluate_with_latency(two_stage_fn, golden_test_set)
print('Baseline:', baseline)
print('Two-stage:', two_stage)

Interpreting NDCG Improvements

Typical improvements from adding cross-encoder re-ranking to dense retrieval range from 0.05 to 0.15 absolute NDCG@5, which translates to roughly 5-15 percentage points. The improvement is larger when: (1) your queries are diverse with many paraphrases, (2) your corpus has many near-relevant chunks, or (3) your first-stage retriever is weak. If you see less than 0.02 NDCG improvement, the benefit may not justify the added complexity.

# Interpreting benchmark results

example_results = {
    'baseline': {'hit_rate@5': 0.78, 'ndcg@5': 0.64, 'p99_latency_ms': 35},
    'two_stage': {'hit_rate@5': 0.89, 'ndcg@5': 0.77, 'p99_latency_ms': 287},
}

delta_ndcg = example_results['two_stage']['ndcg@5'] - example_results['baseline']['ndcg@5']
delta_latency = example_results['two_stage']['p99_latency_ms'] - example_results['baseline']['p99_latency_ms']

print(f'NDCG improvement: +{delta_ndcg:.2f} (+{delta_ndcg/example_results["baseline"]["ndcg@5"]*100:.0f}%)')
print(f'Latency increase: +{delta_latency}ms')
# NDCG improvement: +0.13 (+20%) — clearly worth the 252ms latency cost

End-to-End Answer Quality Measurement

Retrieval metrics measure whether the right documents were retrieved, but the ultimate measure is end-to-end answer quality. Use an LLM judge to evaluate whether answers generated from re-ranked context are more correct and faithful than answers from single-stage context. Score on a 1-5 scale for correctness, faithfulness, and relevance, then average across your test set.

from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = '''
Rate the following answer on a scale of 1-5 for correctness and faithfulness to the context.

Question: {question}
Context: {context}
Answer: {answer}
Ground truth: {ground_truth}

Return a JSON with fields: {"correctness": int, "faithfulness": int, "explanation": str}
'''

def judge_answer(question, context, answer, ground_truth):
    prompt = JUDGE_PROMPT.format(
        question=question, context=context,
        answer=answer, ground_truth=ground_truth,
    )
    response = client.chat.completions.create(
        model='gpt-4o',
        messages=[{'role': 'user', 'content': prompt}],
        response_format={'type': 'json_object'},
    )
    import json
    return json.loads(response.choices[0].message.content)

Stratified Analysis by Query Type

Average metrics hide important differences across query types. Segment your test set into categories — factual lookups (who, what, when), procedural queries (how to), conceptual queries (why, explain), and technical queries (error codes, API names) — and compute metrics separately for each group. Re-ranking often helps most on conceptual and procedural queries where semantic understanding matters more than keyword matching.

def stratified_eval(retriever_fn, test_set, k=5):
    groups = {'factual': [], 'procedural': [], 'conceptual': [], 'technical': []}

    for entry in test_set:
        q = entry['query'].lower()
        if any(w in q for w in ['how to', 'how do', 'steps to']):
            groups['procedural'].append(entry)
        elif any(w in q for w in ['why', 'explain', 'what is the reason']):
            groups['conceptual'].append(entry)
        elif any(c.isupper() for c in q.split()) or 'error' in q:
            groups['technical'].append(entry)
        else:
            groups['factual'].append(entry)

    for group_name, group_entries in groups.items():
        if group_entries:
            metrics = evaluate_pipeline(retriever_fn, group_entries, k)
            print(f'{group_name} ({len(group_entries)} queries): ndcg@{k}={metrics[f"ndcg@{k}"]:.3f}')

Regression Testing with CI Integration

Run your retrieval benchmark as a regression test in CI. Set minimum acceptable thresholds for NDCG@5, MRR, and hit rate. Any pipeline change that causes metrics to drop below the threshold fails the CI build, preventing retrieval quality regressions from shipping to production. This is especially important after changing chunk sizes, embedding models, or re-ranking models.

# pytest integration for retrieval quality gates
import pytest

MIN_NDCG_5 = 0.70
MIN_HIT_RATE_5 = 0.85

def test_retrieval_quality_meets_threshold():
    metrics = evaluate_pipeline(production_retriever_fn, golden_test_set, k=5)
    assert metrics['ndcg@5'] >= MIN_NDCG_5, (
        f'NDCG@5 {metrics["ndcg@5"]:.3f} below threshold {MIN_NDCG_5}'
    )
    assert metrics['hit_rate@5'] >= MIN_HIT_RATE_5, (
        f'Hit rate {metrics["hit_rate@5"]:.3f} below threshold {MIN_HIT_RATE_5}'
    )

# Run with: pytest tests/test_retrieval.py -v

Visualizing Retrieval Metrics

Raw numbers are hard to interpret across multiple experiments. Create a simple comparison table or bar chart that shows NDCG, MRR, hit rate, and latency side by side for baseline, hybrid-only, and hybrid-with-reranking pipelines. Tracking these metrics over time as you make improvements creates a retrieval improvement history that guides future optimization decisions.

def print_comparison_table(results: dict[str, dict]):
    headers = ['Pipeline', 'NDCG@5', 'MRR@5', 'Hit@5', 'P99 ms']
    print('|'.join(f'{h:20}' for h in headers))
    print('-' * (len(headers) * 21))
    for pipeline_name, metrics in results.items():
        row = [
            pipeline_name,
            f'{metrics.get("ndcg@5", 0):.3f}',
            f'{metrics.get("mrr@5", 0):.3f}',
            f'{metrics.get("hit_rate@5", 0):.3f}',
            f'{metrics.get("p99_latency_ms", 0):.0f}',
        ]
        print('|'.join(f'{v:20}' for v in row))

results = {
    'Dense only': {'ndcg@5': 0.64, 'mrr@5': 0.68, 'hit_rate@5': 0.78, 'p99_latency_ms': 35},
    'Hybrid RRF': {'ndcg@5': 0.71, 'mrr@5': 0.74, 'hit_rate@5': 0.84, 'p99_latency_ms': 55},
    'Hybrid + Rerank': {'ndcg@5': 0.77, 'mrr@5': 0.81, 'hit_rate@5': 0.89, 'p99_latency_ms': 287},
}
print_comparison_table(results)

Acting on Benchmark Results

After running benchmarks, use the results to make concrete decisions. If re-ranking improves NDCG by less than 0.03, skip it and focus on improving the first stage. If hit rate is low, your first stage is missing relevant documents — increase the candidate set size or switch to hybrid retrieval. If end-to-end answer quality improves significantly despite modest retrieval gains, the re-ranker may be surfacing highly relevant sentences the LLM uses effectively even if ranked position is unchanged.

Quick Check

Test your understanding of measuring retrieval and re-ranking impact from this lesson.

Lesson Recap

In this lesson you learned: building a golden test set with known relevant documents is essential before measuring retrieval quality, NDCG, MRR, and hit rate are the three core retrieval metrics that together measure ranking quality comprehensively, and end-to-end answer quality using an LLM judge provides the ultimate measure of pipeline improvement. Always run retrieval benchmarks as regression tests in CI. Next up we explore LLM streaming to display tokens as they are generated.

Preguntas frecuentes

¿La lección «Medición del impacto de la reordenación» es gratis?

Sí — el texto completo de «Medición del impacto de la reordenación» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de AI Engineering Academy, actualiza a CoddyKit PRO. El curso de AI Engineering Academy incluye 4 lecciones en total.

¿Qué aprenderé en «Medición del impacto de la reordenación»?

Ejecute una evaluación comparativa antes y después, contrastando la recuperación en una sola etapa con la recuperación en dos etapas y reordenación, y mida NDCG, MRR y la calidad de las respuestas de… Practicas AI Engineering Academy con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.

¿Necesito experiencia previa para empezar AI Engineering Academy?

No se requiere experiencia previa. AI Engineering Academy en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 4 de 4.

¿Cuánto tiempo toma la lección «Medición del impacto de la reordenación»?

La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.

¿Puedo escribir y ejecutar código en esta lección de AI Engineering Academy?

Sí. Cada lección de AI Engineering Academy incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.

Todas las lecciones de este curso

  1. Por qué funciona la recuperación en dos etapas
  2. Reordenación con cross-encoders mediante Cohere y BGE
  3. Compresión contextual y filtrado de relevancia
  4. Medición del impacto de la reordenación
← Volver a AI Engineering Academy