Building a Prompt Test Suite
Organizing tests: golden examples, edge cases, adversarial inputs.
What Is a Prompt Test Suite?
A prompt test suite is a collection of test cases, evaluation tools, and automation that continuously validates your prompts. It is the LLM equivalent of a software project's unit and integration test suite.
A complete suite covers: happy path, edge cases, adversarial inputs, format validation, and regression tests. It runs automatically on each code change and generates a pass/fail report.
Directory Structure
Organize your test suite with a clear directory structure that separates prompts, tests, golden data, and tooling:
# Recommended directory layout
# prompt_project/
# ├── prompts/
# │ ├── sentiment_v3.txt
# │ ├── summarize_v2.txt
# │ └── extract_product_v1.txt
# ├── tests/
# │ ├── conftest.py # shared fixtures
# │ ├── test_sentiment.py
# │ ├── test_summarize.py
# │ └── test_extract.py
# ├── golden_data/
# │ ├── sentiment_tests.json
# │ ├── summarize_tests.json
# │ └── extract_tests.json
# ├── test_results/ # historical test run logs
# │ └── test_history.jsonl
# ├── config/
# │ └── models.json # pinned model versions
# └── pytest.iniAll lessons in this course
- Writing Prompt Test Cases
- Assertion-Based Prompt Testing
- Regression Testing Across Model Updates
- Building a Prompt Test Suite