0Pricing
AI Agents · Lesson

Building a Golden Test Set

Curate 50-200 high-quality (input, expected output) pairs that cover the long tail of real usage.

Building a Golden Test Set is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is a Gold Set?

A "golden test set" (or "gold set") is a curated collection of (input, expected output) pairs that defines what good behavior looks like.

Every change is measured against this set.

Properties of a Good Gold Set

  • Diverse — covers common AND edge cases
  • Curated by humans — not auto-generated
  • Versioned — frozen in your repo
  • Documented — each case has a rationale

How Many Cases?

Rule of thumb:

  • 50 cases — minimum viable
  • 200 cases — good coverage
  • 1000+ cases — mature product

Quality > quantity. 50 well-chosen cases beat 500 random ones.

Sourcing Cases

  1. User interviews — what real users ask
  2. Production logs — questions that came in
  3. Bug reports — every bug becomes an eval
  4. Adversarial generation — ask the LLM to break the agent

Mining Bad Production Traces

for trace in last_week_traces():
    if trace.has_thumbs_down or trace.had_error:
        case = {
            'input': trace.input,
            'why_bad': trace.feedback_comment,
            'expected': None   # to be filled by human reviewer
        }
        save_to_review_queue(case)

Expected Output: Exact vs Heuristic

Two flavors of expected output:

  • Exact match — for extraction, classification, calculation
  • Heuristic — for open-ended Q&A, score whether key facts appear

Heuristic Scoring

def score_answer(actual, expected_facts):
    return sum(1 for f in expected_facts if f.lower() in actual.lower()) / len(expected_facts)

case = {
    'input': 'What is our refund policy?',
    'expected_facts': ['30 days', 'unopened', 'receipt required'],
    'min_score': 0.66   # at least 2 of 3 facts mentioned
}

actual_answer = "You can return items within 30 days if they are unopened."
score = score_answer(actual_answer, case['expected_facts'])
print(f"Score: {score:.2f} (min required: {case['min_score']})")

Adversarial Cases

Include hard cases:

  • Out-of-scope questions ("What is the weather on Mars?")
  • Ambiguous queries
  • Prompt injection attempts
  • Foreign language inputs
  • Very long inputs

Versioning the Gold Set

Store as JSON in your repo. Every PR that updates the set must explain why:

// gold-set.json
[
  {
    "id": "refund-policy-001",
    "input": "What is your refund policy?",
    "expected_facts": ["30 days", "unopened", "receipt required"],
    "added_by": "alice@",
    "added_at": "2025-08-12",
    "reason": "Top customer support question"
  }
]

Test/Holdout Splits

To detect over-fitting, split into:

  • Dev set — iterate on (you see this)
  • Test set — measure on (you do NOT tune to it)
  • Holdout — used only before major releases

Pareto Tagging

Tag cases by category so you can see WHERE you regressed:

tag1 = {'tags': ['policy', 'common-question']}
tag2 = {'tags': ['math', 'edge-case']}
print(tag1)
print(tag2)

# Run eval and report:
print("Category 'policy': 0.92")
print("Category 'math': 0.68   <- regression here")

Eval Rubrics

For complex outputs, write a rubric: what must be present, what must NOT be present, what is preferable:

rubric = {
    'must_include': ['30 days'],
    'must_not_include': ['no refunds'],
    'preferably_includes': ['receipt']
}
for k, v in rubric.items():
    print(f"{k}: {v}")

Source for Cases

What is the highest-signal source of evaluation cases?

Recap

50+ curated cases, sourced from real users and production failures. Version it; tag it; split into dev/test/holdout. Grow it with every bug.

Frequently asked questions

Is the “Building a Golden Test Set” lesson free?

Yes — the full text of “Building a Golden Test Set” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Building a Golden Test Set”?

Curate 50-200 high-quality (input, expected output) pairs that cover the long tail of real usage. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Building a Golden Test Set” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Eval-Driven Development for Agents
  2. Building a Golden Test Set
  3. LLM-as-a-Judge Pitfalls
  4. Benchmark Suites: SWE-Bench, GAIA, ToolBench
← Back to AI Agents