AI Agents · Lesson

Integration Tests for Agent Pipelines

End-to-end tests against real services in isolated test environments.

Integration Tests for Agent Pipelines is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Are Integration Tests for Agents?

Unit tests check individual components in isolation. Integration tests check that multiple components work together correctly in a real or near-real environment.

For agents, this means running the full pipeline — LLM calls, tool execution, data storage — against real or sandboxed services.

End-to-End Test Structure

An end-to-end agent test sends a real query through the entire pipeline and validates the final result. Run these tests in a sandboxed environment — never against your production database or live user data.

import pytest

# Mark as integration test — skipped in fast unit test runs
@pytest.mark.integration
def test_research_agent_full_pipeline():
    from myagent import ResearchAgent

    agent = ResearchAgent(
        openai_api_key='YOUR_TEST_KEY',
        search_api_key='YOUR_TEST_KEY'
    )

    result = agent.run('What is the population of Tokyo?')

    # Structural assertions — not exact string matching
    assert isinstance(result, dict)
    assert result['status'] == 'completed'
    assert 'tokyo' in result['answer'].lower() or 'japan' in result['answer'].lower()
    assert len(result['sources']) >= 1

Test Data Isolation

Integration tests must not pollute shared data. Use dedicated test databases, isolated namespaces, or temporary data that gets cleaned up after the test. Never write test data to production tables.

import os
import pytest

# Use a separate test database URL
@pytest.fixture(scope='session')
def test_db():
    test_db_url = os.environ.get(
        'TEST_DATABASE_URL',
        'postgresql://localhost/myagent_test'  # separate test DB
    )
    # Set up test schema
    from myagent.database import create_tables
    create_tables(test_db_url)
    yield test_db_url
    # Tear down after all tests in the session
    from myagent.database import drop_tables
    drop_tables(test_db_url)

Cleanup After Each Test

Each integration test should clean up any data it created. Use pytest's yield fixture pattern: set up before the yield, clean up after. This ensures tests are independent and can run in any order.

import pytest

@pytest.fixture
def clean_agent_memory(test_db):
    # No setup needed — DB starts empty
    yield
    # Cleanup: delete any records created during this test
    from myagent.database import clear_conversation_history
    clear_conversation_history(test_db)

@pytest.mark.integration
def test_agent_stores_conversation(clean_agent_memory, test_db):
    from myagent import Agent
    agent = Agent(db_url=test_db)

    agent.run('Remember that my name is Alex')
    history = agent.get_history()

    assert len(history) > 0
    assert any('Alex' in str(msg) for msg in history)
    # clean_agent_memory fixture deletes these after the test

Sandboxed Services: Test API Keys

Use dedicated test API keys with limited permissions and quotas for integration tests. Never use production keys in CI. Store test keys as CI environment variables, not in code.

import os
import pytest

# Skip integration tests if test keys are not configured
def requires_integration_keys():
    return pytest.mark.skipif(
        not os.environ.get('OPENAI_TEST_KEY'),
        reason='Integration test keys not configured'
    )

@requires_integration_keys()
@pytest.mark.integration
def test_live_weather_tool():
    from myagent.tools import get_weather
    result = get_weather(city='London', unit='celsius')

    assert result['success'] is True
    assert 'temperature' in result
    assert isinstance(result['temperature'], (int, float))

Using Docker for Sandboxed Databases

For integration tests needing a real database, spin up a Docker container for the test session. This guarantees a clean, isolated database every time and avoids conflicts with your development database.

# conftest.py — docker-based test database
import subprocess
import pytest

@pytest.fixture(scope='session')
def docker_postgres():
    container_id = subprocess.check_output([
        'docker', 'run', '-d',
        '-e', 'POSTGRES_PASSWORD=test',
        '-e', 'POSTGRES_DB=agent_test',
        '-p', '5434:5432',  # use non-standard port to avoid conflicts
        'postgres:15'
    ]).decode().strip()

    import time
    time.sleep(2)  # wait for Postgres to start

    yield 'postgresql://postgres:test@localhost:5434/agent_test'

    subprocess.run(['docker', 'stop', container_id])
    subprocess.run(['docker', 'rm', container_id])

Environment-Specific Test Configs

Integration tests need different configuration for local, CI, and staging environments. Use environment variables and a config helper to select the right settings automatically.

import os

def get_test_config() -> dict:
    env = os.environ.get('TEST_ENV', 'local')

    configs = {
        'local': {
            'db_url': 'postgresql://localhost/agent_test',
            'openai_key': os.environ.get('OPENAI_TEST_KEY', ''),
            'use_real_llm': False  # use mocks locally
        },
        'ci': {
            'db_url': os.environ.get('CI_DATABASE_URL', ''),
            'openai_key': os.environ.get('CI_OPENAI_KEY', ''),
            'use_real_llm': True  # use real LLM in CI integration tests
        },
        'staging': {
            'db_url': os.environ.get('STAGING_DATABASE_URL', ''),
            'openai_key': os.environ.get('STAGING_OPENAI_KEY', ''),
            'use_real_llm': True
        }
    }
    return configs[env]

config = get_test_config()

print(f"TEST_ENV not set -> using '{os.environ.get('TEST_ENV', 'local')}' config")
print(f"DB URL       : {config['db_url']}")
print(f"Use real LLM : {config['use_real_llm']}")

pytest Markers for Test Selection

Use custom pytest markers to categorize tests and run only the relevant subset. Configure markers in pytest.ini and use -m on the command line to select them.

# pytest.ini
# [pytest]
# markers =
#     unit: Fast unit tests with mocked dependencies
#     integration: Slower tests with real or sandboxed services
#     expensive: Tests that make real LLM calls and cost money

# Run only unit tests (fast CI check):
# pytest -m unit

# Run only integration tests:
# pytest -m integration

# Run everything except expensive tests:
# pytest -m 'not expensive'

# In test files:
import pytest

@pytest.mark.unit
def test_tool_format():
    pass  # fast, no external calls

@pytest.mark.integration
@pytest.mark.expensive
def test_with_real_llm():
    pass  # slow, costs tokens

Running Integration Tests in CI

Configure your CI pipeline (GitHub Actions, GitLab CI) to run unit tests on every push and integration tests on a schedule or before releases. This balances speed and coverage.

# .github/workflows/test.yml (abbreviated)
# name: Tests
# on:
#   push:
#     branches: [main, develop]
#   schedule:
#     - cron: '0 2 * * *'  # nightly integration tests
#
# jobs:
#   unit-tests:
#     runs-on: ubuntu-latest
#     steps:
#       - uses: actions/checkout@v3
#       - run: pip install -r requirements.txt
#       - run: pytest -m unit --tb=short
#
#   integration-tests:
#     if: github.event_name == 'schedule'
#     env:
#       CI_OPENAI_KEY: ${{ secrets.CI_OPENAI_KEY }}
#       CI_DATABASE_URL: ${{ secrets.CI_DATABASE_URL }}
#     steps:
#       - run: pytest -m integration --tb=long

print('Unit tests on every push, integration tests nightly')

Measuring Test Coverage for Agents

Use pytest-cov to measure which lines of your agent code are covered by tests. Aim for high coverage of tool functions and agent orchestration logic, even if LLM calls are mocked.

# Install: pip install pytest-cov

# Run tests with coverage report:
# pytest --cov=myagent --cov-report=html -m unit

# This generates an HTML report showing which lines are untested
# Uncovered lines in the agent loop are high-risk areas

# Example coverage config in pyproject.toml:
# [tool.coverage.run]
# omit = ["tests/*", "scripts/*"]
#
# [tool.coverage.report]
# fail_under = 80  # fail if coverage drops below 80%

print('Coverage reports highlight untested code paths in your agent')

Integration Test Best Practices Summary

Key rules for reliable agent integration tests:

Always use separate test databases — never production
Clean up test data after every test with yield fixtures
Use dedicated test API keys with limited quotas
Use pytest markers to separate fast unit tests from slow integration tests
Run unit tests on every push, integration tests on a schedule
Use Docker for disposable, clean database environments

Knowledge Check: Integration Tests

Test your understanding of integration testing for agent pipelines.

Recap: Integration Tests for Agent Pipelines

You now have the knowledge to build a complete, reliable integration test suite:

Use @pytest.mark.integration to separate slow tests from fast unit tests
Isolate test data with dedicated databases and cleanup fixtures
Use Docker or sandboxed services for clean environments
Configure test-specific API keys via environment variables
Run unit tests on every commit, integration tests nightly in CI
Measure coverage with pytest-cov to find untested paths

A well-organized test pyramid keeps your agent reliable as it evolves.

Frequently asked questions

Is the “Integration Tests for Agent Pipelines” lesson free?

Yes — the full text of “Integration Tests for Agent Pipelines” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Integration Tests for Agent Pipelines”?

End-to-end tests against real services in isolated test environments. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Integration Tests for Agent Pipelines” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents