AI Agents · Lesson

Assertion-Based Agent Testing

Checking tool calls, intermediate steps, and final output structure.

Assertion-Based Agent Testing is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Moving Beyond Exact String Matching

Since LLM outputs are non-deterministic, testing them with assert response == 'exact text' is fragile. Instead, write assertions that check the structure and intent of the response without depending on exact wording.

Asserting Tool Calls Were Made

For function-calling agents, the most reliable assertion is verifying that the agent chose to call the right tool. This is structural — it does not depend on the exact wording of the LLM's thought process.

import json
from unittest.mock import patch, MagicMock

@patch('myagent.client.chat.completions.create')
def test_agent_calls_search_tool(mock_create):
    # Mock: agent decides to call search_web
    tool_call = MagicMock()
    tool_call.function.name = 'search_web'
    tool_call.function.arguments = json.dumps({'query': 'Python tutorials'})
    mock_create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(tool_calls=[tool_call]))]
    )

    response = mock_create()  # simulating the agent call
    tc = response.choices[0].message.tool_calls

    assert tc is not None
    assert len(tc) > 0
    assert tc[0].function.name == 'search_web'

# --- demo: give unittest.mock.patch a real dotted path to patch ---
import sys
import types

_myagent = types.ModuleType('myagent')
_myagent.client = types.SimpleNamespace(
    chat=types.SimpleNamespace(completions=types.SimpleNamespace(create=lambda *a, **k: None))
)
sys.modules['myagent'] = _myagent

test_agent_calls_search_tool()
print('test_agent_calls_search_tool: PASS')

Asserting the Correct Tool Name

Beyond just checking that tool calls exist, verify the specific tool name matches expectations. This catches cases where the agent picks the wrong tool for a given query.

import json
from unittest.mock import MagicMock

def extract_tool_calls(response) -> list:
    message = response.choices[0].message
    if not message.tool_calls:
        return []
    return [
        {
            'name': tc.function.name,
            'args': json.loads(tc.function.arguments)
        }
        for tc in message.tool_calls
    ]

# In a test:
# calls = extract_tool_calls(mock_response)
# assert calls[0]['name'] == 'get_weather'
# assert calls[0]['args']['city'] == 'Paris'
print('Tool name and argument assertions are the most reliable agent tests')

Asserting Tool Arguments

After verifying the tool name, check that the arguments are correct. The agent must not only pick the right tool but also populate it with the right parameters from the user's request.

import json
from unittest.mock import patch, MagicMock

@patch('myagent.client.chat.completions.create')
def test_weather_tool_gets_correct_city(mock_create):
    tool_call = MagicMock()
    tool_call.function.name = 'get_weather'
    tool_call.function.arguments = json.dumps({'city': 'Tokyo', 'unit': 'celsius'})
    mock_create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(tool_calls=[tool_call]))]
    )

    response = mock_create()
    args = json.loads(response.choices[0].message.tool_calls[0].function.arguments)

    assert args['city'] == 'Tokyo'
    assert args.get('unit') in ['celsius', 'fahrenheit', None]  # flexible

# --- demo: give unittest.mock.patch a real dotted path to patch ---
import sys
import types

_myagent = types.ModuleType('myagent')
_myagent.client = types.SimpleNamespace(
    chat=types.SimpleNamespace(completions=types.SimpleNamespace(create=lambda *a, **k: None))
)
sys.modules['myagent'] = _myagent

test_weather_tool_gets_correct_city()
print('test_weather_tool_gets_correct_city: PASS')

JSON Schema Validation for Outputs

When your agent returns structured JSON, validate the output against a JSON Schema to ensure all required fields are present and have the correct types. The jsonschema library makes this easy.

# pip install jsonschema
import jsonschema

AGENT_RESPONSE_SCHEMA = {
    'type': 'object',
    'required': ['answer', 'sources', 'confidence'],
    'properties': {
        'answer': {'type': 'string', 'minLength': 1},
        'sources': {
            'type': 'array',
            'items': {'type': 'string', 'format': 'uri'}
        },
        'confidence': {'type': 'number', 'minimum': 0, 'maximum': 1}
    }
}

def test_agent_output_schema(agent_output: dict):
    try:
        jsonschema.validate(instance=agent_output, schema=AGENT_RESPONSE_SCHEMA)
        print('Schema validation passed')
    except jsonschema.ValidationError as e:
        raise AssertionError(f'Invalid agent output: {e.message}')

Keyword Presence Assertions

For text responses where exact wording varies, check that key concepts or words appear in the output. This is flexible yet still meaningful — the agent's answer must at least mention the relevant terms.

def assert_keywords_present(text: str, keywords: list, require_all: bool = True):
    lower_text = text.lower()
    found = [kw.lower() in lower_text for kw in keywords]

    if require_all:
        missing = [kw for kw, f in zip(keywords, found) if not f]
        assert not missing, f'Missing keywords: {missing}'
    else:
        assert any(found), f'None of {keywords} found in: {text[:100]}'

# Tests
response = 'The capital city of France is Paris, located in western Europe.'
assert_keywords_present(response, ['paris', 'france', 'capital'])
print('All keywords present!')  # passes

assert_keywords_present(response, ['spain', 'france'], require_all=False)
print('At least one keyword present!')  # passes

Asserting Response Format: Type Checks

Type assertions are fast and reliable. Verify that the agent returns a dict (not None), that list fields are lists, and that numeric fields are in valid ranges.

def test_agent_returns_valid_structure(agent_result):
    # Type checks
    assert isinstance(agent_result, dict), 'Result must be a dict'
    assert isinstance(agent_result.get('answer'), str), 'answer must be a string'
    assert isinstance(agent_result.get('steps'), list), 'steps must be a list'

    # Non-empty checks
    assert len(agent_result['answer']) > 0, 'answer must not be empty'
    assert len(agent_result['steps']) >= 1, 'must have at least one step'

    # Range checks
    confidence = agent_result.get('confidence', 0)
    assert 0.0 <= confidence <= 1.0, 'confidence must be 0-1'

print('Structural assertions are fast and reliable')

Asserting finish_reason

The finish_reason field tells you why the model stopped generating. Asserting it helps detect issues: 'stop' means a clean answer, 'tool_calls' means the agent wants to call a tool, 'length' means truncation.

from unittest.mock import MagicMock

def test_agent_stops_cleanly(mock_response):
    finish_reason = mock_response.choices[0].finish_reason
    assert finish_reason in ('stop', 'tool_calls'), \
        f'Unexpected finish_reason: {finish_reason}'

def test_no_truncation(mock_response):
    finish_reason = mock_response.choices[0].finish_reason
    assert finish_reason != 'length', \
        'Response was truncated — increase max_tokens'

# Example mock for a clean stop
mock = MagicMock()
mock.choices = [MagicMock(finish_reason='stop')]
test_agent_stops_cleanly(mock)
print('finish_reason: stop — clean termination')

Asserting the Number of Steps in a Loop

An agent that runs in a loop should complete in a reasonable number of steps. Assert that the agent finishes within a maximum iteration count — this catches infinite loops that your max_iterations guard is meant to prevent.

def test_agent_completes_in_bounded_steps(mock_agent):
    result = mock_agent.run('Search for the weather in Paris')

    # Agent should complete within 5 steps
    assert result['steps_taken'] <= 5, \
        f'Agent took too many steps: {result["steps_taken"]}'

    # Agent should produce a final answer, not exit on timeout
    assert result['status'] == 'completed', \
        f'Agent did not complete: {result["status"]}'

    assert result['answer'] is not None

print('Bounding step count prevents runaway agents from passing tests')

Parametrize Tests for Multiple Inputs

pytest's @pytest.mark.parametrize lets you run the same test with many different inputs. This is ideal for testing that your agent routes different query types to the correct tools.

import pytest
from unittest.mock import patch, MagicMock
import json

@pytest.mark.parametrize('query,expected_tool', [
    ('What is the weather in Tokyo?', 'get_weather'),
    ('Calculate 15% of 200', 'calculator'),
    ('Search for Python books', 'web_search'),
    ('What time is it in Berlin?', 'get_time'),
])
@patch('myagent.client.chat.completions.create')
def test_agent_tool_routing(mock_create, query, expected_tool):
    tool_call = MagicMock()
    tool_call.function.name = expected_tool
    tool_call.function.arguments = json.dumps({'input': query})
    mock_create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(tool_calls=[tool_call]))]
    )
    response = mock_create()
    actual = response.choices[0].message.tool_calls[0].function.name
    assert actual == expected_tool

Writing Custom Assertion Helpers

As your agent test suite grows, extract common assertion patterns into helpers. This makes tests shorter, more readable, and easier to maintain when the agent's response format changes.

import json

def assert_tool_called(response, tool_name: str, required_args: dict = None):
    message = response.choices[0].message
    assert message.tool_calls, 'Expected tool call but got plain text'
    names = [tc.function.name for tc in message.tool_calls]
    assert tool_name in names, f'Expected {tool_name}, got {names}'

    if required_args:
        for tc in message.tool_calls:
            if tc.function.name == tool_name:
                args = json.loads(tc.function.arguments)
                for key, val in required_args.items():
                    assert args.get(key) == val, \
                        f'Arg {key}: expected {val}, got {args.get(key)}'

# Clean test using the helper:
# assert_tool_called(response, 'get_weather', {'city': 'Paris'})

# --- demo ---
from unittest.mock import MagicMock

tool_call = MagicMock()
tool_call.function.name = 'get_weather'
tool_call.function.arguments = json.dumps({'city': 'Paris'})
response = MagicMock(choices=[MagicMock(message=MagicMock(tool_calls=[tool_call]))])

assert_tool_called(response, 'get_weather', {'city': 'Paris'})
print('assert_tool_called passed: agent called get_weather with city=Paris')

Knowledge Check: Assertion-Based Agent Testing

Test your understanding of assertion strategies for agent tests.

Recap: Assertion-Based Agent Testing

You now have a complete assertion toolkit for agent tests:

Check that tool_calls is non-empty when the agent should use a tool
Assert the correct tool name with tc.function.name == 'expected_tool'
Validate tool arguments by parsing tc.function.arguments as JSON
Use jsonschema.validate() for structured output validation
Use keyword presence checks for flexible text assertions
Check finish_reason and step counts for loop agents
Use @pytest.mark.parametrize for multiple input scenarios

Frequently asked questions

Is the “Assertion-Based Agent Testing” lesson free?

Yes — the full text of “Assertion-Based Agent Testing” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Assertion-Based Agent Testing”?

Checking tool calls, intermediate steps, and final output structure. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Assertion-Based Agent Testing” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents