0Pricing
AI Prompt Engineering · Lesson

Assertion-Based Prompt Testing

Checking outputs with contains(), regex, JSON schema, and LLM-as-judge.

Assertion-Based Prompt Testing is a free AI Prompt Engineering lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Assertions for LLM Outputs

Assertion-based testing applies to LLMs the same principle used in unit testing: make explicit claims about what the output must contain or not contain, and fail immediately when the claim is violated.

Unlike unit tests with deterministic functions, LLM assertions deal with probabilistic text output — requiring more flexible assertion types: contains, matches_schema, satisfies_regex, llm_judge_score_above.

Basic Assertions: contains and not_contains

The simplest assertions check for keyword presence or absence. These work well for classification tasks, structured outputs, and safety checks.

import openai
client = openai.OpenAI(api_key='sk-...')

def call_prompt(system, user, temperature=0):
    resp = client.chat.completions.create(
        model='gpt-4o',
        messages=[
            {'role': 'system', 'content': system},
            {'role': 'user', 'content': user}
        ],
        temperature=temperature
    )
    return resp.choices[0].message.content

# Keyword presence assertion
def assert_contains(output, keyword, case_sensitive=False):
    text = output if case_sensitive else output.lower()
    kw = keyword if case_sensitive else keyword.lower()
    assert kw in text, f'Expected "{keyword}" in output, got: {output[:100]}'

# Keyword absence assertion
def assert_not_contains(output, forbidden, case_sensitive=False):
    text = output if case_sensitive else output.lower()
    kw = forbidden if case_sensitive else forbidden.lower()
    assert kw not in text, f'Forbidden "{forbidden}" found in output: {output[:100]}'

JSON Schema Validation

When your prompt is supposed to return structured JSON, validate the output against a schema. A schema validation failure means the prompt has a format problem — either the model added prose, or the JSON structure is wrong.

import json
from jsonschema import validate, ValidationError

PRODUCT_SCHEMA = {
    'type': 'object',
    'properties': {
        'name': {'type': 'string'},
        'price': {'type': 'number', 'minimum': 0},
        'available': {'type': 'boolean'}
    },
    'required': ['name', 'price', 'available'],
    'additionalProperties': False
}

def assert_valid_json_schema(output, schema):
    try:
        data = json.loads(output.strip())
    except json.JSONDecodeError as e:
        raise AssertionError(f'Output is not valid JSON: {e}\nOutput: {output[:200]}')
    try:
        validate(instance=data, schema=schema)
    except ValidationError as e:
        raise AssertionError(f'JSON does not match schema: {e.message}\nOutput: {output[:200]}')
    return data

# Test
output = call_prompt(
    'Extract product info as JSON: {"name": ..., "price": ..., "available": ...}',
    'Widget Pro costs $49.99 and is in stock.'
)
product = assert_valid_json_schema(output, PRODUCT_SCHEMA)
print('Parsed product:', product)

Regex Matching

Regex assertions validate output format precisely — useful for outputs that should follow a specific pattern like dates, phone numbers, or structured codes.

import re

def assert_matches_regex(output, pattern, flags=0):
    if not re.search(pattern, output, flags):
        raise AssertionError(
            f'Output does not match pattern /{pattern}/\nOutput: {output[:200]}'
        )

def assert_output_is_label(output, valid_labels):
    cleaned = output.strip().upper()
    assert cleaned in valid_labels, (
        f'Expected one of {valid_labels}, got: {repr(cleaned)}'
    )

# Examples
output = call_prompt('Classify sentiment as POSITIVE, NEGATIVE, or NEUTRAL:', 'Great product!')
assert_output_is_label(output, {'POSITIVE', 'NEGATIVE', 'NEUTRAL'})

date_output = call_prompt('Extract the date in YYYY-MM-DD format:', 'Meeting on November 15, 2024')
assert_matches_regex(date_output, r'^\d{4}-\d{2}-\d{2}$')

LLM-as-Judge Scoring

For open-ended outputs, use a second LLM call to evaluate quality. This is called LLM-as-judge. The judge model receives the original prompt, the output, and evaluation criteria, then returns a score.

def llm_judge_score(original_prompt, output, criteria, max_score=10):
    judge_prompt = (
        f'Evaluate the following AI response on a scale of 1-{max_score}.\n'
        f'Evaluation criteria: {criteria}\n\n'
        f'Original prompt: {original_prompt}\n\n'
        f'AI response: {output}\n\n'
        f'Return only a number from 1 to {max_score}.'
    )
    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': judge_prompt}],
        temperature=0
    )
    score_text = resp.choices[0].message.content.strip()
    return int(score_text)

def assert_llm_score_above(original_prompt, output, criteria, min_score=7):
    score = llm_judge_score(original_prompt, output, criteria)
    assert score >= min_score, f'LLM judge score {score} < minimum {min_score}'

Using pytest for Prompt Tests

pytest is the standard Python testing framework and works well for prompt tests. Each test function corresponds to one test case. pytest collects, runs, and reports them automatically.

# test_sentiment_prompt.py
import pytest
import openai

client = openai.OpenAI(api_key='sk-...')
SYSTEM_PROMPT = 'Classify the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. Return only the label.'

def classify(text):
    resp = client.chat.completions.create(
        model='gpt-4o',
        messages=[
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': text}
        ],
        temperature=0
    )
    return resp.choices[0].message.content.strip().upper()

# pytest automatically discovers functions starting with test_
def test_positive_sentiment():
    assert classify('I love this product!') == 'POSITIVE'

def test_negative_sentiment():
    assert classify('Terrible experience.') == 'NEGATIVE'

def test_neutral_sentiment():
    assert classify('It arrived on time.') == 'NEUTRAL'

# Run: pytest test_sentiment_prompt.py -v

Parameterized Tests in pytest

Use @pytest.mark.parametrize to run the same test function across many inputs without repeating code. This is the cleanest way to build a comprehensive test suite.

# test_sentiment_parametrized.py
import pytest

TEST_CASES = [
    ('I love this!', 'POSITIVE'),
    ('Worst purchase ever.', 'NEGATIVE'),
    ('It works.', 'NEUTRAL'),
    ('Amazing!', 'POSITIVE'),
    ('Terrible!', 'NEGATIVE'),
    ('OK I guess.', 'NEUTRAL'),
]

@pytest.mark.parametrize('text,expected', TEST_CASES)
def test_sentiment_classification(text, expected):
    result = classify(text)
    assert result == expected, f'For "{text}": expected {expected}, got {result}'

# pytest test_sentiment_parametrized.py -v
# Output shows each test case individually:
# PASSED test_sentiment_parametrized.py::test_sentiment_classification[I love this!-POSITIVE]
# PASSED test_sentiment_parametrized.py::test_sentiment_classification[Worst purchase ever.-NEGATIVE]

Fixtures for Shared Prompt State

Use pytest fixtures to share expensive setup across tests — like loading a prompt template or creating an API client once per test session.

# conftest.py — fixtures available to all test files in the directory
import pytest
import openai

@pytest.fixture(scope='session')
def llm_client():
    return openai.OpenAI(api_key='sk-...')

@pytest.fixture(scope='session')
def sentiment_prompt():
    with open('prompts/sentiment_v3.txt') as f:
        return f.read()

# test_sentiment.py
def test_positive_with_fixture(llm_client, sentiment_prompt):
    resp = llm_client.chat.completions.create(
        model='gpt-4o',
        messages=[
            {'role': 'system', 'content': sentiment_prompt},
            {'role': 'user', 'content': 'I love this!'}
        ],
        temperature=0
    )
    assert 'POSITIVE' in resp.choices[0].message.content.upper()

Handling Flaky Tests

LLM outputs are probabilistic — even at temperature=0, different model deployments or versions may produce different outputs. Handle flakiness with retry logic and tolerance thresholds.

import pytest

def run_with_retry(fn, n=3):
    '''Run fn up to n times, pass if any run succeeds.'''
    failures = []
    for _ in range(n):
        try:
            fn()
            return  # passed
        except AssertionError as e:
            failures.append(str(e))
    raise AssertionError(f'Failed all {n} attempts. Last: {failures[-1]}')

def test_positive_with_retry():
    def check():
        result = classify('I love this!')
        assert result == 'POSITIVE'
    run_with_retry(check, n=3)

# Or use pytest-retry plugin:
# @pytest.mark.flaky(reruns=3)
# def test_positive_sentiment():
#     assert classify('I love this!') == 'POSITIVE'

Test Performance and Cost

Each test case is an API call — for 100 test cases at $0.005/call = $0.50 per full test run. Strategies to manage cost:

  • Cache responses for static test inputs and run from cache in CI
  • Run the full suite nightly; run only a smoke test subset (10 cases) on each PR
  • Use a cheaper model (gpt-4o-mini) for most tests; run on gpt-4o only for regression suite
import hashlib, json

RESPONSE_CACHE = {}

def cached_classify(text, use_cache=True):
    key = hashlib.md5(text.encode()).hexdigest()
    if use_cache and key in RESPONSE_CACHE:
        return RESPONSE_CACHE[key]
    result = classify(text)
    RESPONSE_CACHE[key] = result
    return result

# Persist cache to disk for CI
def load_cache(path='test_cache.json'):
    global RESPONSE_CACHE
    try:
        with open(path) as f:
            RESPONSE_CACHE = json.load(f)
    except FileNotFoundError:
        RESPONSE_CACHE = {}

def save_cache(path='test_cache.json'):
    with open(path, 'w') as f:
        json.dump(RESPONSE_CACHE, f, indent=2)

Test Output Reports

pytest produces detailed reports that highlight which test cases failed and why. Use pytest --tb=short -v for concise failure messages. For CI, use --junitxml to produce JUnit XML reports compatible with GitHub Actions, GitLab CI, and Jenkins.

# Run test suite and generate reports
# In terminal:
# pytest tests/prompt/ -v --tb=short --junitxml=test_results.xml

# In Python (for programmatic use):
import subprocess

def run_prompt_tests(test_dir='tests/prompt'):
    result = subprocess.run(
        ['pytest', test_dir, '-v', '--tb=short', '--junitxml=test_results.xml'],
        capture_output=True, text=True
    )
    print(result.stdout)
    if result.returncode != 0:
        print('TESTS FAILED')
        print(result.stderr)
    return result.returncode == 0

passed = run_prompt_tests()

Knowledge Check

When would you use LLM-as-judge scoring instead of an exact match assertion in prompt testing?

Recap: Assertion-Based Prompt Testing

Key assertion types for LLM outputs:

  • contains / not_contains: keyword presence — good for labels and safety checks
  • JSON schema validation: validates structured output format
  • Regex matching: validates specific patterns (dates, codes)
  • LLM-as-judge: evaluates open-ended prose quality

Use pytest with @pytest.mark.parametrize for clean, scalable test suites. Cache responses to manage cost. Run a smoke subset on each PR; full suite nightly. Next lesson: regression testing across model updates.

Frequently asked questions

Is the “Assertion-Based Prompt Testing” lesson free?

Yes — the full text of “Assertion-Based Prompt Testing” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Assertion-Based Prompt Testing”?

Checking outputs with contains(), regex, JSON schema, and LLM-as-judge. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Assertion-Based Prompt Testing” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Writing Prompt Test Cases
  2. Assertion-Based Prompt Testing
  3. Regression Testing Across Model Updates
  4. Building a Prompt Test Suite
← Back to AI Prompt Engineering