0Pricing
AI Prompt Engineering · Lesson

Evaluating DSPy Pipelines

Metrics, dev sets, and the evaluate() function for automated assessment.

Evaluating DSPy Pipelines is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Evaluation Matters in DSPy

DSPy optimization is only as good as your evaluation. A weak metric produces a compiled program that scores well on that metric but fails in production. A proper evaluation harness lets you compare unoptimized vs optimized programs and catch regressions when you update your pipeline.

The dspy.Evaluate Class

dspy.Evaluate runs your program over a dataset, applies a metric, and reports aggregate scores. It supports parallelism via num_threads for fast evaluation over large datasets.

import dspy

# Build a devset of labeled examples
devset = [
    dspy.Example(question='What is 7 * 8?', answer='56').with_inputs('question'),
    dspy.Example(question='Name the largest planet.', answer='Jupiter').with_inputs('question'),
    # ... more examples
]

# Create evaluator
evaluate = dspy.Evaluate(
    devset=devset,
    metric=exact_match_metric,  # Your metric function
    num_threads=4,              # Parallel evaluation
    display_progress=True,      # Show progress bar
    display_table=True,         # Show per-example results
)

# Run
score = evaluate(my_program)
print(f'Overall score: {score:.1%}')

Writing Metric Functions

Metric functions have the signature (example, prediction, trace=None) -> float. They compare the program's prediction against the ground truth in the example.

The trace parameter is non-None during optimization (not evaluation) — you can use it to apply different logic in compilation vs evaluation.

import dspy

def exact_match_metric(example, prediction, trace=None):
    return float(
        example.answer.strip().lower() == prediction.answer.strip().lower()
    )

def contains_metric(example, prediction, trace=None):
    """Check if expected answer appears anywhere in prediction."""
    return float(example.answer.lower() in prediction.answer.lower())

def length_penalized_metric(example, prediction, trace=None):
    """Reward correct answers, penalize overly long ones."""
    correct = float(example.answer.lower() in prediction.answer.lower())
    length_ok = float(len(prediction.answer.split()) <= 20)
    return correct * (0.8 + 0.2 * length_ok)

# Use any of these as the metric parameter
evaluate = dspy.Evaluate(devset=devset, metric=contains_metric)

Pass/Fail Threshold Patterns

For binary metrics, you can define a threshold: a prediction is 'passing' if it meets a minimum quality bar. This is useful for filtering few-shot demos during bootstrap optimization.

import dspy

def quality_metric(example, prediction, trace=None):
    """
    Multi-factor metric with pass/fail threshold.
    Returns float 0.0 to 1.0.
    During compilation (trace is not None), DSPy uses this to decide
    which traces to bootstrap as demos.
    """
    score = 0.0

    # Factor 1: Factual correctness (0.6 weight)
    if example.answer.lower() in prediction.answer.lower():
        score += 0.6

    # Factor 2: Conciseness (0.4 weight)
    word_count = len(prediction.answer.split())
    if word_count <= 15:
        score += 0.4
    elif word_count <= 30:
        score += 0.2

    # During optimization: only use examples scoring >= 0.6
    if trace is not None:
        return score >= 0.6

    return score

Splitting Data: Train, Dev, Test

Follow standard ML data splitting practices in DSPy:

  • Trainset: Used by optimizer to bootstrap demos (20-200 examples)
  • Devset: Used by optimizer for validation during search
  • Testset: Held out entirely — only used for final evaluation
import random

# All labeled examples
all_examples = load_examples()  # Returns list of dspy.Example
random.shuffle(all_examples)

total = len(all_examples)
train_end = int(total * 0.6)
dev_end   = int(total * 0.8)

trainset = all_examples[:train_end]    # 60% for optimization
devset   = all_examples[train_end:dev_end]  # 20% for validation
testset  = all_examples[dev_end:]      # 20% held out

print(f'Train: {len(trainset)}, Dev: {len(devset)}, Test: {len(testset)}')

Comparing Optimized vs Unoptimized

Always benchmark your compiled program against the baseline (uncompiled) program on the same test set. This proves the optimization actually helped and quantifies the improvement.

import dspy

evaluate = dspy.Evaluate(
    devset=testset,
    metric=exact_match_metric,
    num_threads=4,
    display_progress=True,
)

# Baseline: unoptimized program
baseline_score = evaluate(unoptimized_program)
print(f'Baseline (no optimization): {baseline_score:.1%}')

# BootstrapFewShot compiled
bs_score = evaluate(bootstrap_compiled_program)
print(f'BootstrapFewShot compiled:  {bs_score:.1%}')

# MIPRO compiled
mipro_score = evaluate(mipro_compiled_program)
print(f'MIPRO compiled:             {mipro_score:.1%}')

# Pick the winner
print(f'Best improvement: +{max(bs_score, mipro_score) - baseline_score:.1%}')

Parallelism with num_threads

Large evaluation sets would take hours sequentially. num_threads in dspy.Evaluate runs predictions in parallel, cutting wall-clock time proportionally.

Match num_threads to your API rate limits — too many threads triggers rate-limit errors.

import dspy
import time

devset = [...]  # 200 examples

# Sequential evaluation
start = time.time()
evaluate_seq = dspy.Evaluate(devset=devset, metric=metric, num_threads=1)
score_seq = evaluate_seq(program)
print(f'Sequential: {time.time()-start:.0f}s')

# Parallel evaluation (4 threads)
start = time.time()
evaluate_par = dspy.Evaluate(devset=devset, metric=metric, num_threads=4)
score_par = evaluate_par(program)
print(f'Parallel (4 threads): {time.time()-start:.0f}s')
# Typically ~4x faster — same score, less wait time

Interpreting Evaluation Output

When display_table=True, DSPy shows a detailed table with each example, the prediction, and whether it passed the metric. This is invaluable for diagnosing failure patterns.

Look for: systematic failures on a question type, metric edge cases, or examples your training set doesn't cover.

import dspy

evaluate = dspy.Evaluate(
    devset=devset,
    metric=exact_match_metric,
    num_threads=2,
    display_progress=True,
    display_table=10,  # Show first 10 rows of results table
    return_outputs=True,  # Return (score, outputs) tuple
)

score, outputs = evaluate(program, return_all_scores=True)

# Find failing examples
failures = [
    (ex, pred, s)
    for ex, pred, s in outputs
    if s == 0.0
]
print(f'Failures: {len(failures)}/{len(devset)}')
for ex, pred, _ in failures[:3]:
    print(f'Q: {ex.question}')
    print(f'Expected: {ex.answer}')
    print(f'Got: {pred.answer}')

Using LLM-Graded Metrics

For open-ended outputs where exact match fails, use an LLM to grade quality. DSPy makes this easy — your metric function can itself call a DSPy predictor.

import dspy

class GradeAnswer(dspy.Signature):
    """Grade whether the predicted answer is correct given the reference."""
    question: str = dspy.InputField()
    reference_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    is_correct: bool = dspy.OutputField(
        desc='True if the predicted answer is semantically correct'
    )

grader = dspy.Predict(GradeAnswer)

def llm_graded_metric(example, prediction, trace=None):
    result = grader(
        question=example.question,
        reference_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return float(result.is_correct)

# Use this metric when answers can vary in phrasing
evaluate = dspy.Evaluate(devset=devset, metric=llm_graded_metric)

Regression Testing with Evaluation

Treat your DSPy evaluation suite like a test suite. Every time you update your signature, module architecture, or training data, re-run evaluation and compare scores to catch regressions.

import json
import dspy

def run_and_save_evaluation(program, program_name, testset, metric):
    evaluate = dspy.Evaluate(
        devset=testset,
        metric=metric,
        num_threads=4,
    )
    score = evaluate(program)

    # Save score to history file
    history_file = 'eval_history.json'
    try:
        with open(history_file) as f:
            history = json.load(f)
    except FileNotFoundError:
        history = []

    history.append({'program': program_name, 'score': score})
    with open(history_file, 'w') as f:
        json.dump(history, f, indent=2)

    print(f'{program_name}: {score:.1%}')
    return score

Evaluation Best Practices

Key evaluation principles for DSPy pipelines:

  • Keep your testset strictly held out — never optimize on it
  • Use at least 50-100 test examples for reliable scores
  • Match your metric to your actual production goal
  • Compare multiple optimizers — results vary by task
  • Track scores over time to detect regressions
  • Inspect failures manually to improve your training data

Knowledge Check: Metric Function Trace Parameter

In a DSPy metric function, what does a non-None trace parameter indicate?

Recap: Evaluating DSPy Pipelines

dspy.Evaluate runs your program over a labeled devset, applies a metric function, and reports aggregate scores. Metric functions follow the pattern (example, prediction, trace=None) -> float. Use num_threads for parallel evaluation, and display_table=True to diagnose failures. Always compare optimized vs unoptimized programs on a held-out testset. For open-ended outputs, LLM-graded metrics outperform exact string matching.

Frequently asked questions

Is the “Evaluating DSPy Pipelines” lesson free?

Yes — the full text of “Evaluating DSPy Pipelines” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Evaluating DSPy Pipelines”?

Metrics, dev sets, and the evaluate() function for automated assessment. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Evaluating DSPy Pipelines” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Introduction to DSPy Framework
  2. Defining Signatures and Modules
  3. Compiling and Optimizing Prompts
  4. Evaluating DSPy Pipelines
← Back to AI Prompt Engineering