AI Prompt Engineering · Lesson

Repair and Validation Loops

Fixing malformed output.

Repair and Validation Loops is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Repair Loops Exist

Even with schemas, output can fail: truncated JSON from a token cap, hallucinated extra fields, a string where a number was required, or a violated business rule. A repair loop detects the failure and feeds it back to the model to correct.

It is the safety net that turns a 98% success rate into a 99.9% one.

The Validate-Repair Cycle

The canonical loop: generate, validate, and if invalid, re-prompt with the specific error, up to a bounded number of attempts.

def generate_valid(prompt, schema, max_tries=3):
    msgs = [{'role': 'user', 'content': prompt}]
    for _ in range(max_tries):
        out = call_model(msgs)
        ok, err = validate(out, schema)
        if ok:
            return out
        msgs.append({'role': 'assistant', 'content': out})
        msgs.append({'role': 'user',
                     'content': 'Invalid. Fix this error and resend JSON only: ' + err})
    raise ValueError('repair budget exhausted')

Specific Errors Beat Generic Ones

Repair quality depends on feedback precision. 'Invalid JSON' helps little; 'Field rating must be an integer between 1 and 5, got 7' almost always fixes it in one turn.

Pipe your validator's exact error message (path plus reason) back to the model.

def validate(obj, schema):
    v = jsonschema.Draft202012Validator(schema)
    errs = sorted(v.iter_errors(obj), key=lambda e: e.path)
    if not errs:
        return True, None
    msg = '; '.join(f"{list(e.path)}: {e.message}" for e in errs)
    return False, msg

Layered Validation

Validation has tiers; check cheapest first:

Syntactic — does it parse as JSON?
Structural — does it match the schema?
Semantic — do values satisfy business rules and cross-field invariants?

A discount cannot exceed the subtotal even if both are valid numbers; that is a semantic check.

def semantic_ok(o):
    return o['discount'] <= o['subtotal'] and o['total'] == o['subtotal'] - o['discount']

Deterministic Pre-Repair

Before spending a model call, try cheap deterministic fixes for common defects:

Strip markdown code fences.
Extract the outermost JSON object from surrounding prose.
Remove trailing commas.

Only escalate to a model repair if deterministic cleanup fails.

def extract_json(text):
    text = text.strip().removeprefix('json').strip()
    start, end = text.find('{'), text.rfind('}')
    if start != -1 and end != -1:
        return text[start:end + 1]
    return text

Handling Truncation

If finish_reason == 'length', the JSON is cut off; repairing the fragment is futile. Instead, raise max_tokens, reduce the requested payload, or stream and request a continuation. Distinguish truncation from malformed output, because they need opposite responses.

if resp.choices[0].finish_reason == 'length':
    # do NOT feed truncated JSON back; regenerate with more room
    return retry_with(max_tokens=resp_max * 2)

Bound the Loop

Unbounded repair is a cost and latency hazard, and a stuck model can loop forever. Always cap attempts, add a per-request budget, and define a fallback (degrade gracefully, queue for human review, or return a safe default).

try:
    data = generate_valid(prompt, schema, max_tries=3)
except ValueError:
    log_for_review(prompt)
    data = SAFE_DEFAULT

Repair Inside Tool Calls

For function calling, the repair channel is the tool result. Return a structured error and the model self-corrects on the next turn, no separate loop needed.

msgs.append({'role': 'tool', 'tool_call_id': call.id,
  'content': json.dumps({'error': 'amount must be positive', 'got': args['amount']})})
# next model turn will re-issue the call with a corrected argument

Idempotency and Side Effects

When a repair loop wraps an action (not just generation), ensure retries are idempotent. Use idempotency keys so a re-issued call does not double-charge or duplicate a record. Validate before committing side effects, never after.

Observe Repair Rates

The repair rate is a leading quality signal. A rising rate means a prompt regression, a schema that is too strict, or model drift. Log attempts-per-success and the top error paths, and alert when they spike.

metrics.histogram('llm.repair.attempts', tries)
metrics.increment('llm.repair.error', tags={'path': top_error_path})

Combine Constraints with Repair

The strongest pipeline layers defenses: constrained decoding or strict schema mode to minimize failures, deterministic cleanup for cheap fixes, and a bounded model-repair loop for the rare residual. No single layer is sufficient alone.

Quick Check

The API returns finish_reason length with cut-off JSON. What is the correct response?

Recap

Robust repair pipelines:

Validate in tiers: syntactic, structural, semantic.
Feed precise, path-level errors back to the model.
Try deterministic cleanup before paying for a model call.
Detect truncation separately and regenerate.
Bound attempts, ensure idempotency, and monitor repair rates.

You have completed structured generation. Next course: guardrails and output validation.

Frequently asked questions

Is the “Repair and Validation Loops” lesson free?

Yes — the full text of “Repair and Validation Loops” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Repair and Validation Loops”?

Fixing malformed output. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Repair and Validation Loops” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering