Writing Prompt Test Cases
Input-expected_output pairs: the unit test of prompt engineering.
Why Prompt Testing Needs Formal Test Cases
Informal prompt testing — 'I tried it a few times and it worked' — fails to catch edge cases, regressions after model updates, and failures on unusual inputs. Formal test cases bring software engineering discipline to prompt development: every test is explicit, repeatable, and automatically evaluated.
Anatomy of a Prompt Test Case
A prompt test case has three components:
- Input: the prompt with all variables filled in — the exact string sent to the model
- Expected: a specification of what constitutes a correct response (not necessarily the exact output, but criteria)
- Evaluator: a function that takes the actual output and returns a pass/fail signal
from dataclasses import dataclass
from typing import Callable, Any
@dataclass
class PromptTestCase:
name: str
input_prompt: str # The full prompt sent to the model
expected_criteria: str # Human-readable description of expected behavior
evaluator: Callable[[str], bool] # Returns True if output passes
# Example test case
test = PromptTestCase(
name='sentiment_positive',
input_prompt='Classify the sentiment: I love this product!',
expected_criteria='Response must contain POSITIVE',
evaluator=lambda output: 'POSITIVE' in output.upper()
)All lessons in this course
- Writing Prompt Test Cases
- Assertion-Based Prompt Testing
- Regression Testing Across Model Updates
- Building a Prompt Test Suite