AI Engineering Academy · Lesson

Building an Automated Evaluation Harness

Create a repeatable evaluation pipeline that runs your full RAG system against a test set, computes all metrics, and generates a report so you can track improvements over time.

Building an Automated Evaluation Harness is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is an Evaluation Harness?

An evaluation harness is a repeatable, automated pipeline that runs your full RAG system against a standardized test set, computes all metrics, and produces a report. The key word is repeatable: every time you make a change to your chunking strategy, embedding model, prompt, or LLM, you run the same harness and compare results against a baseline. This transforms RAG development from subjective tinkering into data-driven engineering.

Harness Architecture

A well-designed evaluation harness has four layers: Test data management (load and version the golden dataset), pipeline execution (run each test question through the full RAG pipeline), metrics computation (calculate all retrieval and generation metrics), and report generation (save results with version info and produce a diff against the previous baseline). Each layer should be independently testable and configurable.

class RAGEvaluationHarness:
    def __init__(self, retriever, llm_client, config):
        self.retriever = retriever
        self.llm_client = llm_client
        self.config = config  # chunk_size, top_k, model, threshold, etc.
        self.results = []

    def run(self, golden_dataset):
        for item in golden_dataset:
            result = self._evaluate_single(item)
            self.results.append(result)
        metrics = self._compute_metrics()
        self._save_report(metrics)
        return metrics

Running Each Test Case

For each question in the golden dataset, run the complete RAG pipeline and capture all intermediate outputs: the retrieved chunk IDs and scores, the formatted context, the generated answer, and the token count. Storing these intermediate values is essential for debugging failures — when a question scores poorly you can inspect exactly which chunks were retrieved and why the answer was wrong without re-running the expensive pipeline.

import time

def _evaluate_single(self, item):
    start = time.perf_counter()
    query_vector = embed_query(item['question'])
    chunks = self.retriever.retrieve(query_vector, top_k=self.config['top_k'])
    filtered_chunks = filter_by_score(chunks, self.config['threshold'])
    context = format_context(filtered_chunks)
    answer_result = generate_answer(item['question'], context, self.llm_client)
    latency_ms = (time.perf_counter() - start) * 1000

    return {
        'question': item['question'],
        'expected_answer': item['answer'],
        'generated_answer': answer_result['answer'],
        'retrieved_chunk_ids': [c['id'] for c in filtered_chunks],
        'retrieved_scores': [c['score'] for c in filtered_chunks],
        'relevant_chunk_ids': item['relevant_chunk_ids'],
        'context_texts': [c['text'] for c in filtered_chunks],
        'tokens_used': answer_result['tokens_used'],
        'latency_ms': round(latency_ms)
    }

Computing All Metrics in One Pass

After collecting all test case outputs, compute the full suite of metrics in a single pass over the results. Separate retrieval metrics (computed from chunk IDs) from generation metrics (computed by calling the judge LLM). Batch the judge LLM calls to maximize efficiency — group faithfulness evaluations and send them in parallel with asyncio rather than sequentially. Log progress since generation metrics can take several minutes for 100+ test cases.

def _compute_metrics(self):
    # Retrieval metrics (no LLM calls needed)
    hit_rates = []
    mrr_scores = []
    for r in self.results:
        retrieved = r['retrieved_chunk_ids']
        relevant = set(r['relevant_chunk_ids'])
        hit = any(rid in relevant for rid in retrieved)
        hit_rates.append(1.0 if hit else 0.0)
        for rank, rid in enumerate(retrieved, 1):
            if rid in relevant:
                mrr_scores.append(1.0 / rank)
                break
        else:
            mrr_scores.append(0.0)

    metrics = {
        'hit_rate_at_5': sum(hit_rates) / len(hit_rates),
        'mrr': sum(mrr_scores) / len(mrr_scores),
        'mean_latency_ms': sum(r['latency_ms'] for r in self.results) / len(self.results),
        'mean_tokens': sum(r['tokens_used'] for r in self.results) / len(self.results)
    }
    return metrics

Saving Results with Version Information

Every evaluation run should be saved with version metadata so you can compare results across configurations. Include the git commit hash of the code, the configuration parameters (embedding model, chunk size, K, threshold, LLM model), the timestamp, and a human-readable run description. Store results in a JSON Lines file or a database table. This creates a permanent history of how your system has evolved.

import json
import subprocess
from datetime import datetime

def _save_report(self, metrics):
    git_hash = subprocess.check_output(
        ['git', 'rev-parse', '--short', 'HEAD']
    ).decode().strip()

    report = {
        'run_id': datetime.utcnow().strftime('%Y%m%d_%H%M%S'),
        'git_commit': git_hash,
        'config': self.config,
        'metrics': metrics,
        'n_test_cases': len(self.results),
        'timestamp': datetime.utcnow().isoformat()
    }

    with open('eval_history.jsonl', 'a') as f:
        f.write(json.dumps(report) + '\n')
    print(f'Saved evaluation run: {report["run_id"]}')
    print(json.dumps(metrics, indent=2))

Comparing Against Baseline

After each run, automatically compare against the previous baseline and flag regressions. A regression is any metric that drops by more than a threshold (e.g., 2 percentage points). Print a diff table showing the metric changes. If any metric regresses significantly, the evaluation run should fail with a non-zero exit code, which will cause a CI/CD pipeline to block the deployment of that change.

def compare_to_baseline(current_metrics, baseline_file='best_eval.json'):
    import json
    from pathlib import Path
    if not Path(baseline_file).exists():
        print('No baseline yet. Saving current as baseline.')
        Path(baseline_file).write_text(json.dumps(current_metrics, indent=2))
        return True

    baseline = json.loads(Path(baseline_file).read_text())
    regressions = []
    print('\nMetric comparison (current vs baseline):')
    for metric, current_val in current_metrics.items():
        baseline_val = baseline.get(metric, 0)
        delta = current_val - baseline_val
        status = 'OK' if delta >= -0.02 else 'REGRESSION'
        print(f'  {metric}: {current_val:.3f} vs {baseline_val:.3f} ({delta:+.3f}) {status}')
        if status == 'REGRESSION':
            regressions.append(metric)
    return len(regressions) == 0

Integrating into CI/CD

The evaluation harness is most powerful when integrated into your CI/CD pipeline. Configure it to run automatically on every pull request that modifies chunking logic, embedding model configuration, prompt templates, or retrieval parameters. The pipeline passes only if all metrics meet the minimum thresholds and no metric regresses from the main branch baseline. This prevents accidental quality regressions from shipping to production.

# GitHub Actions workflow (eval.yml)
# on:
#   pull_request:
#     paths:
#       - 'rag/**'
#       - 'prompts/**'
#       - 'config/**'
# jobs:
#   evaluate:
#     runs-on: ubuntu-latest
#     steps:
#       - uses: actions/checkout@v3
#       - name: Install dependencies
#         run: pip install -r requirements.txt
#       - name: Run evaluation harness
#         run: |
#           python eval/run_harness.py \
#             --test-set eval/golden_dataset.json \
#             --config config/rag_config.yaml \
#             --fail-on-regression
#         env:
#           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Generating Human-Readable Reports

In addition to raw metric files, generate a human-readable HTML or Markdown report that your team can review in pull request comments. Include a summary table of all metrics, a list of failed test cases with the question, expected answer, generated answer, and which chunks were retrieved, and a trend chart showing metrics over the last 10 runs. Visual reports help non-technical stakeholders understand whether the system is improving.

def generate_markdown_report(metrics, failed_cases, run_id):
    lines = [
        f'# RAG Evaluation Report — {run_id}\n',
        '## Summary Metrics',
        '| Metric | Score | Target |',
        '|--------|-------|--------|',
        f'| Hit Rate@5 | {metrics["hit_rate_at_5"]:.1%} | > 80% |',
        f'| MRR | {metrics["mrr"]:.3f} | > 0.70 |',
        f'| Mean Latency | {metrics["mean_latency_ms"]:.0f}ms | < 500ms |',
        '',
        f'## Failed Cases ({len(failed_cases)} failures)'
    ]
    for case in failed_cases[:10]:  # show first 10
        lines += [
            f'**Q:** {case["question"]}',
            f'**Expected:** {case["expected_answer"]}',
            f'**Generated:** {case["generated_answer"]}\n'
        ]
    return '\n'.join(lines)

Tracking Cost Per Evaluation Run

Evaluation runs cost money — they call the embedding API, the LLM API, and the judge LLM. Track the cost of each evaluation run alongside the quality metrics. A comprehensive evaluation of 100 test cases typically costs $0.50 to $2.00 depending on the models used. Use cheaper models for judge calls (GPT-4o-mini for faithfulness scoring) and reserve expensive models for generation. Include estimated run cost in the saved report so you can budget evaluation into your development cycle.

def estimate_run_cost(results, config):
    # Embedding cost
    embed_tokens = sum(len(r['question'].split()) * 1.3 for r in results)
    embed_cost = (embed_tokens / 1_000_000) * 0.02  # $0.02/1M tokens

    # Generation cost
    total_gen_tokens = sum(r['tokens_used'] for r in results)
    gen_cost = (total_gen_tokens / 1_000_000) * 5.0  # gpt-4o approx

    # Judge cost (faithfulness evals)
    judge_cost = len(results) * 0.001  # ~$0.001 per eval with gpt-4o-mini

    total = embed_cost + gen_cost + judge_cost
    print(f'Evaluation cost estimate: ${total:.2f}')
    print(f'  Embedding: ${embed_cost:.3f}')
    print(f'  Generation: ${gen_cost:.3f}')
    print(f'  Judgment: ${judge_cost:.3f}')
    return total

Scheduled Evaluation for Production Monitoring

Beyond CI/CD evaluation on code changes, run the harness on a schedule in production — daily or weekly — testing against real user queries sampled from your logs. This detects data drift: as the document corpus evolves and user query patterns shift, system quality can degrade without any code change. Schedule weekly evaluation runs that sample 50 recent user queries, evaluate them, and send a quality digest to your team's Slack channel automatically.

# Example scheduled evaluation (cron job or scheduled cloud function)
import random

def sample_production_queries(query_log_file, n=50):
    with open(query_log_file) as f:
        all_queries = [json.loads(line) for line in f]
    sample = random.sample(all_queries, min(n, len(all_queries)))
    # Convert to golden dataset format (without expected answers — use LLM judge)
    return [
        {'question': q['user_question'], 'relevant_chunk_ids': []}
        for q in sample
    ]

# Run weekly evaluation against production queries
if __name__ == '__main__':
    prod_queries = sample_production_queries('/var/log/rag_queries.jsonl')
    harness = RAGEvaluationHarness(retriever, llm_client, config)
    metrics = harness.run(prod_queries)
    send_slack_digest(metrics)

Visualizing Metric Trends Over Time

Raw numbers in a JSONL file are hard to interpret at a glance. Build a simple trend visualization that plots each metric over the last 20 evaluation runs on a line chart. Use the run timestamp as the x-axis and metric score as the y-axis. Plot a horizontal line at the minimum acceptable threshold. When a metric dips below the threshold line, the issue is immediately visible without reading through raw data. Tools like Matplotlib or a simple web dashboard (Grafana, Streamlit) work well for this.

import json
import matplotlib.pyplot as plt
from pathlib import Path

def plot_metric_trends(history_file='eval_history.jsonl', metric='hit_rate_at_5'):
    records = [
        json.loads(line)
        for line in Path(history_file).read_text().strip().split('\n')
    ]
    timestamps = [r['timestamp'][:10] for r in records[-20:]]
    scores = [r['metrics'].get(metric, 0) for r in records[-20:]]
    plt.figure(figsize=(10, 4))
    plt.plot(timestamps, scores, marker='o', label=metric)
    plt.axhline(y=0.80, color='r', linestyle='--', label='Min threshold')
    plt.title(f'{metric} over last 20 evaluations')
    plt.xticks(rotation=45)
    plt.tight_layout()
    plt.savefig(f'eval_trend_{metric}.png')
    print(f'Saved trend chart for {metric}')

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: how to structure a complete evaluation harness with test data management, pipeline execution, metrics computation, and report generation, how to compare runs against a baseline and fail CI/CD on regressions, how to generate human-readable reports for team review, and how to run scheduled production monitoring to detect data drift without code changes. You now have a complete foundation for building and evaluating production RAG systems.

Frequently asked questions

Is the “Building an Automated Evaluation Harness” lesson free?

Yes — the full text of “Building an Automated Evaluation Harness” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Building an Automated Evaluation Harness”?

Create a repeatable evaluation pipeline that runs your full RAG system against a test set, computes all metrics, and generates a report so you can track improvements over time. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Building an Automated Evaluation Harness” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Why Evaluation Matters in RAG
Retrieval Metrics: Hit Rate, MRR, and NDCG
Generation Metrics: Faithfulness and Answer Relevance
Building an Automated Evaluation Harness

← Back to AI Engineering Academy