AI Prompt Engineering · Lesson

Assertion-Based Prompt Testing

Checking outputs with contains(), regex, JSON schema, and LLM-as-judge.

Assertions for LLM Outputs

Assertion-based testing applies to LLMs the same principle used in unit testing: make explicit claims about what the output must contain or not contain, and fail immediately when the claim is violated.

Unlike unit tests with deterministic functions, LLM assertions deal with probabilistic text output — requiring more flexible assertion types: contains, matches_schema, satisfies_regex, llm_judge_score_above.

Basic Assertions: contains and not_contains

The simplest assertions check for keyword presence or absence. These work well for classification tasks, structured outputs, and safety checks.

import openai
client = openai.OpenAI(api_key='sk-...')

def call_prompt(system, user, temperature=0):
    resp = client.chat.completions.create(
        model='gpt-4o',
        messages=[
            {'role': 'system', 'content': system},
            {'role': 'user', 'content': user}
        ],
        temperature=temperature
    )
    return resp.choices[0].message.content

# Keyword presence assertion
def assert_contains(output, keyword, case_sensitive=False):
    text = output if case_sensitive else output.lower()
    kw = keyword if case_sensitive else keyword.lower()
    assert kw in text, f'Expected "{keyword}" in output, got: {output[:100]}'

# Keyword absence assertion
def assert_not_contains(output, forbidden, case_sensitive=False):
    text = output if case_sensitive else output.lower()
    kw = forbidden if case_sensitive else forbidden.lower()
    assert kw not in text, f'Forbidden "{forbidden}" found in output: {output[:100]}'

All lessons in this course

← Back to AI Prompt Engineering