AI Prompt Engineering · Lesson

Root Cause Analysis for Prompts

Isolating whether failure is in context, instruction, format, or model capability.

Root Cause Analysis for Prompts is a free AI Prompt Engineering lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Root Cause Analysis Matters

When a prompt fails, there are multiple possible causes. Randomly tweaking wording wastes time and may fix the symptom without fixing the root cause — leading to the same failure on slightly different inputs.

Root cause analysis (RCA) is a systematic process to isolate the specific reason a prompt fails, so the fix addresses the actual problem.

The Four Root Cause Categories

Every prompt failure traces back to one of four root causes:

Context problem: the model lacks information needed to answer correctly
Instruction ambiguity: the instruction has multiple valid interpretations, and the model chose the wrong one
Format conflict: two parts of the prompt give contradictory formatting instructions
Model capability limit: the task requires reasoning or knowledge beyond what this model can reliably do

Root Cause 1: Context Problem

A context problem occurs when the model answers incorrectly because the prompt does not provide the information needed. The model fills the gap with training data — which may be outdated, wrong, or hallucinated.

Test: provide the missing information directly in the prompt and see if the answer improves. If yes, the fix is to add context (e.g., via RAG retrieval).

# Failing prompt — no context
prompt_v1 = 'What is the current price of our Pro plan?'

# Context problem test: inject the information
prompt_v2 = '''
Our pricing (as of today):
- Free: $0/month
- Pro: $19/month
- Enterprise: $99/month

Question: What is the current price of our Pro plan?
'''
# If v2 succeeds and v1 fails -> root cause is context problem

Root Cause 2: Instruction Ambiguity

Instruction ambiguity occurs when the prompt can be reasonably interpreted in multiple ways and the model picks the wrong interpretation.

Example: 'Summarize briefly' — does 'briefly' mean one sentence, one paragraph, or three bullet points? The model guesses. Test: replace the ambiguous phrase with a precise specification and check if failure resolves.

# Ambiguous
prompt_ambiguous = 'Summarize the following article briefly.'

# Precise — ambiguity removed
prompt_precise = (
    'Summarize the following article in exactly 2 sentences. '
    'Do not exceed 50 words. Output only the summary, nothing else.'
)

# Test: if precise version succeeds, root cause was ambiguity
# Fix: replace vague qualifiers with exact specifications

Root Cause 3: Format Conflict

A format conflict occurs when two parts of the prompt give contradictory instructions. The model must choose one and ignore the other — usually choosing the more recent or more prominent instruction.

Example: system prompt says 'respond in plain text', user message says 'use markdown'. The model may follow either, inconsistently.

# Format conflict example
system_prompt = 'You are a helpful assistant. Always respond in plain text without any formatting.'

user_message = 'List the top 5 benefits of exercise. Use markdown bullet points.'

# The model faces a conflict: plain text vs markdown.
# Detection: if output format is inconsistent across runs, look for conflicting instructions.

# Fix: ensure all format instructions agree. Move format to system prompt only.
system_prompt_fixed = (
    'You are a helpful assistant. '
    'Always respond using markdown bullet points for lists.'
)

Root Cause 4: Model Capability Limit

A capability limit failure occurs when the task genuinely exceeds what the model can do reliably. This is different from the other three causes — no prompt change will fully fix it.

Signs: the failure rate is high even with clear instructions and full context. Fixes: use a more powerful model, break the task into simpler steps, or add a verification step.

# Capability limit test: try the same task on different models
models = ['gpt-4o-mini', 'gpt-4o', 'gpt-4o-2024-11-20']
results = {}

for model in models:
    resp = client.chat.completions.create(
        model=model,
        messages=[{'role': 'user', 'content': complex_reasoning_prompt}]
    )
    results[model] = evaluate(resp.choices[0].message.content)

# If accuracy improves with more capable models -> capability limit
for model, score in results.items():
    print(f'{model}: {score:.0%} accuracy')

The Isolation Method

To identify which root cause is active, use systematic isolation:

Run the failing prompt and classify the failure type (wrong answer, format, etc.)
Add information → if fixed: context problem
Clarify instructions → if fixed: ambiguity
Audit for contradictions → if fixed: format conflict
Upgrade model → if fixed: capability limit

Only one fix should be needed. If multiple fixes are needed, there were multiple root causes.

def rca_test(base_prompt, test_input, expected_output):
    results = {}

    # Test 1: base (failing) prompt
    results['base'] = run_and_evaluate(base_prompt, test_input, expected_output)

    # Test 2: add context
    results['with_context'] = run_and_evaluate(
        base_prompt + '\nContext: ' + get_context(test_input),
        test_input, expected_output
    )

    # Test 3: clarify instructions
    results['clarified'] = run_and_evaluate(
        clarify(base_prompt), test_input, expected_output
    )

    for name, passed in results.items():
        print(f'{name}: {"PASS" if passed else "FAIL"}')

Elimination vs Confirmation

RCA follows two modes:

Elimination: rule out causes that are NOT responsible (test each hypothesis and see which one does NOT change the output)
Confirmation: identify the cause that, when fixed, consistently resolves the failure across multiple test inputs

Confirmation requires at least 3 test inputs. A fix that works on one input but not others has not solved the root cause — it may have solved a symptom.

def confirm_root_cause(fix_fn, test_cases, threshold=0.9):
    '''fix_fn: a function that takes a prompt and returns a fixed prompt'''
    passed = 0
    for case in test_cases:
        fixed_prompt = fix_fn(case['prompt'])
        result = run_and_evaluate(fixed_prompt, case['input'], case['expected'])
        if result:
            passed += 1

    pass_rate = passed / len(test_cases)
    print(f'Fix pass rate: {pass_rate:.0%}')
    if pass_rate >= threshold:
        print('Root cause CONFIRMED — fix is reliable.')
    else:
        print('Root cause NOT confirmed — failure has multiple causes.')

Documenting the Root Cause

After identifying the root cause, document it in the prompt change log. Include:

Failure type observed
Root cause category
Hypothesis tested
Evidence (which test passed after the fix)
The specific change made to the prompt

This prevents future engineers from re-investigating the same failure and enables pattern recognition across prompts.

rca_record = {
    'prompt_id': 'summarize_v3',
    'failure_type': 'wrong_format',
    'root_cause': 'format_conflict',
    'hypothesis': 'System prompt said plain text, user message asked for markdown',
    'evidence': 'Removing markdown instruction from user message resolved failure on 8/8 test cases',
    'fix_applied': 'Moved all format instructions to system prompt; removed format instructions from user template',
    'fix_date': '2024-11-15'
}

Common Mistake: Fixing the Wrong Cause

The most common RCA mistake is fixing the symptom rather than the cause. Example:

Symptom: model returns JSON with extra prose prefix
Wrong fix: add post-processing to strip prose from output
Actual root cause: no format instruction in prompt + model defaults to conversational style
Correct fix: add explicit 'Return only valid JSON. No other text.' instruction

Post-processing hacks accumulate technical debt. Root-cause fixes are durable.

RCA for Intermittent Failures

Some failures are intermittent — the prompt works 80% of the time but fails 20%. These are harder to diagnose because running the prompt once gives a correct result.

Approach: run the prompt 10–20 times on the same input. If failure rate is non-zero, the prompt has a probabilistic root cause — usually instruction ambiguity or high temperature. Fix: make the instruction more specific, or reduce temperature.

def measure_failure_rate(prompt, test_input, expected, runs=20):
    failures = 0
    for _ in range(runs):
        resp = client.chat.completions.create(
            model='gpt-4o',
            messages=[{'role': 'user', 'content': prompt + '\n' + test_input}],
            temperature=0.7
        )
        if not evaluate(resp.choices[0].message.content, expected):
            failures += 1
    print(f'Failure rate: {failures}/{runs} = {failures/runs:.0%}')

Knowledge Check

A prompt asks for a JSON response but occasionally returns JSON preceded by a conversational prefix like 'Sure! Here is the JSON:'. After adding 'Return only valid JSON. No other text.' the failure stops. What was the root cause?

Recap: Root Cause Analysis

The four root cause categories for prompt failures:

Context problem: model lacks needed information — fix: add context via RAG or direct injection
Instruction ambiguity: vague instruction with multiple interpretations — fix: add precision
Format conflict: contradictory formatting instructions — fix: consolidate in system prompt
Model capability limit: task exceeds model ability — fix: upgrade model or decompose task

Use systematic isolation to test each hypothesis. Confirm the fix across multiple test cases. Document findings. Next lesson: systematic binary search debugging.

Frequently asked questions

Is the “Root Cause Analysis for Prompts” lesson free?

Yes — the full text of “Root Cause Analysis for Prompts” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Root Cause Analysis for Prompts”?

Isolating whether failure is in context, instruction, format, or model capability. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Root Cause Analysis for Prompts” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering