AI Prompt Engineering · Lesson

Regression Testing Across Model Updates

Running test suites when upgrading from GPT-4 to GPT-4o or Claude 3 to 3.5.

Regression Testing Across Model Updates is a free AI Prompt Engineering lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Model Updates Break Prompts

LLM providers regularly update their models: GPT-4 → GPT-4o → GPT-4o-2024-11-20, Claude 3 → Claude 3.5 → Claude 3.7. Each update changes the model's behavior — often improving most tasks but occasionally regressing on specific prompts.

Without a test suite, regressions are invisible until users report them. With a test suite, you detect regressions within minutes of a model update.

The Model Update Problem

Model updates can cause three types of changes:

Improvements: previously failing test cases now pass — good
Neutral: behavior unchanged — most tests
Regressions: previously passing test cases now fail — must investigate

Even a 1% regression rate is significant: if you have 200 test cases and 2 start failing after a model update, those 2 may be your most critical use cases.

MODEL_HISTORY = [
    {'model': 'gpt-4', 'deployed': '2023-03-14', 'pass_rate': 0.87},
    {'model': 'gpt-4-turbo', 'deployed': '2023-11-06', 'pass_rate': 0.91},
    {'model': 'gpt-4o', 'deployed': '2024-05-13', 'pass_rate': 0.93},
    {'model': 'gpt-4o-2024-11-20', 'deployed': '2024-11-20', 'pass_rate': None},  # to be measured
]

# Goal: measure pass_rate for the new model before deploying to production

Running the Test Suite on a New Model

When a new model version is announced, run your full test suite against both the old and new model. Compare pass rates. Identify which specific test cases changed behavior.

import openai
client = openai.OpenAI(api_key='sk-...')

def run_suite_on_model(test_cases, system_prompt, model):
    results = []
    for test in test_cases:
        resp = client.chat.completions.create(
            model=model,
            messages=[
                {'role': 'system', 'content': system_prompt},
                {'role': 'user', 'content': test['input']}
            ],
            temperature=0
        )
        output = resp.choices[0].message.content
        passed = test['evaluator'](output)
        results.append({'id': test['id'], 'passed': passed, 'output': output})
    pass_rate = sum(r['passed'] for r in results) / len(results)
    return results, pass_rate

old_results, old_rate = run_suite_on_model(TEST_CASES, PROMPT, 'gpt-4o')
new_results, new_rate = run_suite_on_model(TEST_CASES, PROMPT, 'gpt-4o-2024-11-20')
print(f'Old: {old_rate:.1%} | New: {new_rate:.1%} | Delta: {(new_rate-old_rate):+.1%}')

Diffing Results Between Model Versions

After running both suites, identify cases that changed status: passed → failed (regressions) and failed → passed (improvements).

def diff_results(old_results, new_results):
    old_by_id = {r['id']: r for r in old_results}
    new_by_id = {r['id']: r for r in new_results}

    regressions = []
    improvements = []

    for test_id, new_r in new_by_id.items():
        old_r = old_by_id.get(test_id)
        if old_r is None:
            continue
        if old_r['passed'] and not new_r['passed']:
            regressions.append({'id': test_id, 'old_output': old_r['output'], 'new_output': new_r['output']})
        elif not old_r['passed'] and new_r['passed']:
            improvements.append({'id': test_id})

    print(f'Regressions: {len(regressions)}')
    print(f'Improvements: {len(improvements)}')
    for reg in regressions:
        print(f'  REGRESSED: {reg["id"]}')
        print(f'    Old: {reg["old_output"][:60]}')
        print(f'    New: {reg["new_output"][:60]}')
    return regressions, improvements

regressions, improvements = diff_results(old_results, new_results)

Tracking Model Version in Test Logs

Every test run log must include the exact model identifier — not just 'gpt-4o' but the full versioned string 'gpt-4o-2024-11-20'. OpenAI and Anthropic frequently update the model behind an alias (e.g., 'gpt-4o') without changing the alias name, making the log essential for debugging historical behavior.

import openai, json
from datetime import datetime, timezone

client = openai.OpenAI(api_key='sk-...')

def run_test_with_version_tracking(test, system_prompt, model_alias):
    resp = client.chat.completions.create(
        model=model_alias,
        messages=[
            {'role': 'system', 'content': system_prompt},
            {'role': 'user', 'content': test['input']}
        ],
        temperature=0
    )
    output = resp.choices[0].message.content
    return {
        'test_id': test['id'],
        'model_alias': model_alias,
        'model_actual': resp.model,  # The exact versioned model ID returned by the API
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'output': output,
        'passed': test['evaluator'](output)
    }

Investigating Regressions

When a regression is found, investigate before updating the prompt. Ask: did the prompt become worse, or did the model become better in a way that changed behavior?

Example: GPT-4o's successor might follow format instructions more strictly, causing a test to fail because the old test expected a slightly non-compliant output. In this case, the model improved and the test needs updating — not the prompt.

def investigate_regression(test_id, old_output, new_output, prompt, user_input):
    # Step 1: check if old behavior was actually wrong
    judge_prompt = (
        f'Given this task prompt: {prompt}\n'
        f'And this user input: {user_input}\n\n'
        f'Which output is better?\n'
        f'A) {old_output}\n'
        f'B) {new_output}\n\n'
        'Reply with A or B and one sentence explanation.'
    )
    resp = client.chat.completions.create(
        model='gpt-4o',
        messages=[{'role': 'user', 'content': judge_prompt}],
        temperature=0
    )
    verdict = resp.choices[0].message.content
    print(f'Judge verdict for {test_id}: {verdict}')
    # If judge says B (new output) is better: update the test, not the prompt

When to Update the Prompt vs the Test

After a regression investigation, choose one of three actions:

Update the prompt: the new model requires different instruction phrasing to achieve the same behavior
Update the test: the old expected output was wrong or suboptimal; the new output is actually better
Accept the regression: the new model cannot do this task reliably; use the old model alias for this use case

REGRESSION_ACTIONS = {
    'prompt_updated': {
        'when': 'New model ignores instruction the old model followed. Same behavior needed.',
        'action': 'Add stronger/rephrased instruction. Re-run tests on new model.'
    },
    'test_updated': {
        'when': 'New model output is objectively better but fails old test criteria.',
        'action': 'Update expected output in test. Document the improvement.'
    },
    'model_pinned': {
        'when': 'New model cannot perform this task reliably despite prompt changes.',
        'action': 'Pin the model alias to the old versioned ID for this endpoint.'
    }
}

Pinning Model Versions

When a model update causes unacceptable regressions, pin the model to the previous versioned ID while you investigate. Do not use an alias (e.g., 'gpt-4o') for production — always use a specific version.

# Bad practice: alias can point to different model versions over time
client.chat.completions.create(
    model='gpt-4o',  # could be any version at any time
    messages=[...]
)

# Good practice: pin to a specific version for production
client.chat.completions.create(
    model='gpt-4o-2024-11-20',  # guaranteed behavior
    messages=[...]
)

# Track in config
MODEL_CONFIG = {
    'sentiment_classifier': 'gpt-4o-2024-11-20',
    'code_generator': 'gpt-4o-2024-11-20',
    'summarizer': 'gpt-4o-mini-2024-07-18',
}

Automated Regression in CI

Run the regression test suite automatically when the model version in your config changes. Use GitHub Actions or similar CI to detect regressions before deployment.

# .github/workflows/prompt_regression.yml (YAML, shown as comment)
# name: Prompt Regression Tests
# on:
#   push:
#     paths:
#       - 'config/models.json'  # triggers when model version changes
#       - 'prompts/*.txt'       # triggers when prompts change
# jobs:
#   test:
#     runs-on: ubuntu-latest
#     steps:
#       - uses: actions/checkout@v4
#       - name: Install dependencies
#         run: pip install pytest openai jsonschema
#       - name: Run prompt test suite
#         run: pytest tests/prompts/ -v --junitxml=results.xml
#         env:
#           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

# In Python: read model version from config
import json
with open('config/models.json') as f:
    MODEL_CONFIG = json.load(f)
MODEL = MODEL_CONFIG['sentiment_classifier']

Handling Cross-Provider Regressions

Regression testing also applies when switching providers: GPT-4o → Claude, Claude → Gemini. The same test suite reveals which prompts require changes for the new provider's behavior differences.

def cross_provider_test(test_cases, system_prompt, providers):
    all_results = {}
    for provider, config in providers.items():
        results, rate = run_suite_on_model(
            test_cases, system_prompt, model=config['model']
        )
        all_results[provider] = {'rate': rate, 'results': results}
        print(f'{provider}: {rate:.1%}')

    # Find cases that fail on one provider but not another
    for test in test_cases:
        outcomes = {
            p: next(r for r in all_results[p]['results'] if r['id'] == test['id'])['passed']
            for p in providers
        }
        if not all(outcomes.values()):
            failing = [p for p, passed in outcomes.items() if not passed]
            print(f'  {test["id"]}: FAILS on {failing}')

    return all_results

Monitoring Pass Rate Trends

Plot pass rate over time by reading from the test history log. A declining trend signals prompt degradation or model drift. A sudden drop signals a regression event (model update or prompt change).

import json

def plot_pass_rate_trend(history_file='test_history.jsonl'):
    runs = []
    with open(history_file) as f:
        for line in f:
            runs.append(json.loads(line))

    runs.sort(key=lambda r: r['run_id'])

    print('Pass rate trend:')
    for run in runs[-10:]:  # last 10 runs
        bar = '#' * int(run['pass_rate'] * 40)
        print(f"{run['run_id'][:10]} {run['model']:30} {run['pass_rate']:.0%} |{bar}|")

    # Detect regression
    if len(runs) >= 2:
        delta = runs[-1]['pass_rate'] - runs[-2]['pass_rate']
        if delta < -0.05:
            print(f'ALERT: pass rate dropped {delta:.0%} since last run!')

plot_pass_rate_trend()

Knowledge Check

When a new model version causes a test to fail, and investigation reveals that the new model's output is actually better than the old output, what is the correct action?

Recap: Regression Testing Across Model Updates

Key practices for regression testing when models update:

Run full test suite on both old and new model — compare pass rates and diff specific failures
Track exact model version in every test log — not just the alias
Investigate each regression: is it a prompt issue, a test issue, or a model capability issue?
Pin models in production to specific versioned IDs, not aliases
Automate in CI — trigger on model config or prompt file changes

Next lesson: building a complete prompt test suite with tooling support.

Frequently asked questions

Is the “Regression Testing Across Model Updates” lesson free?

Yes — the full text of “Regression Testing Across Model Updates” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Regression Testing Across Model Updates”?

Running test suites when upgrading from GPT-4 to GPT-4o or Claude 3 to 3.5. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Regression Testing Across Model Updates” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering