AI Agents · Lesson

Assertion-Based Agent Testing

Checking tool calls, intermediate steps, and final output structure.

Moving Beyond Exact String Matching

Since LLM outputs are non-deterministic, testing them with assert response == 'exact text' is fragile. Instead, write assertions that check the structure and intent of the response without depending on exact wording.

Asserting Tool Calls Were Made

For function-calling agents, the most reliable assertion is verifying that the agent chose to call the right tool. This is structural — it does not depend on the exact wording of the LLM's thought process.

import json
from unittest.mock import patch, MagicMock

@patch('myagent.client.chat.completions.create')
def test_agent_calls_search_tool(mock_create):
    # Mock: agent decides to call search_web
    tool_call = MagicMock()
    tool_call.function.name = 'search_web'
    tool_call.function.arguments = json.dumps({'query': 'Python tutorials'})
    mock_create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(tool_calls=[tool_call]))]
    )

    response = mock_create()  # simulating the agent call
    tc = response.choices[0].message.tool_calls

    assert tc is not None
    assert len(tc) > 0
    assert tc[0].function.name == 'search_web'

All lessons in this course

← Back to AI Agents