AI Engineering Academy · Lesson

Validating and Retrying Bad Outputs

Implement a validation layer that checks extracted data against business rules, automatically retries with corrective feedback when validation fails, and logs failure patterns.

Validating and Retrying Bad Outputs is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why LLM Outputs Need Validation

Even with structured outputs and Pydantic schemas, LLM extraction can produce outputs that are syntactically valid but semantically wrong. A confidence score of 1.5 (outside the 0-1 range), a price of -99.99, a date string that cannot be parsed, or a phone number with letters — all of these pass JSON parsing but fail your business rules.

Validation is a separate concern from extraction. Extraction asks: 'Did we get structured data?' Validation asks: 'Is the structured data correct and usable?' Both layers are necessary for a production-grade pipeline. Think of it as a two-stage filter: the LLM extracts, your validator accepts or rejects.

Layers of Validation

A robust output validation system operates at multiple levels:

Schema validation (Pydantic): correct field types, required fields present, enums match allowed values — handled automatically by structured outputs
Format validation: phone numbers match a regex, emails are valid, dates are parseable, amounts are within realistic ranges
Business logic validation: invoice total equals sum of line items, end date is after start date, quantity is a positive integer
Cross-field validation: a field's value depends on another field's value (e.g., discount percent cannot exceed 100)
Semantic validation: extracted company name matches a known company in your database

Pydantic Validators for Format Checks

Pydantic's field_validator decorator lets you add custom validation logic that runs when the model is instantiated. Use this for format-level checks like regex validation of phone numbers and emails, date parsing, and range checks on numeric fields.

from pydantic import BaseModel, Field, field_validator
from typing import Optional
import re
from datetime import datetime

class ExtractedInvoice(BaseModel):
    vendor: str
    invoice_number: Optional[str]
    amount: float = Field(gt=0, description='Must be positive')
    currency: str = Field(min_length=3, max_length=3)
    invoice_date: str

    @field_validator('currency')
    @classmethod
    def currency_must_be_uppercase(cls, v):
        return v.upper()

    @field_validator('invoice_date')
    @classmethod
    def parse_date(cls, v):
        # Try to parse common date formats
        for fmt in ('%Y-%m-%d', '%d/%m/%Y', '%m/%d/%Y', '%B %d, %Y'):
            try:
                datetime.strptime(v, fmt)
                return v
            except ValueError:
                continue
        raise ValueError(f'Cannot parse date: {v}')

    @field_validator('amount')
    @classmethod
    def reasonable_amount(cls, v):
        if v > 10_000_000:
            raise ValueError(f'Amount {v} seems unreasonably large. Flag for review.')
        return round(v, 2)

The Retry with Corrective Feedback Pattern

When validation fails, the most effective recovery strategy is retry with corrective feedback: send the validation error message back to the model as context, explaining what went wrong and asking it to fix only the failing fields. This gives the model the information it needs to correct its output rather than just re-trying blind.

import openai
from pydantic import BaseModel, ValidationError, Field

client = openai.OpenAI()

class PriceExtraction(BaseModel):
    product: str
    price_usd: float = Field(gt=0, lt=100000)
    quantity: int = Field(ge=1)

def extract_with_retry(text: str, max_retries: int = 3) -> PriceExtraction:
    messages = [
        {'role': 'system', 'content': 'Extract product pricing information.'},
        {'role': 'user', 'content': text}
    ]

    for attempt in range(max_retries):
        result = client.beta.chat.completions.parse(
            model='gpt-4o-mini',
            messages=messages,
            response_format=PriceExtraction
        )
        msg = result.choices[0].message
        if msg.refusal:
            raise ValueError(f'Model refused: {msg.refusal}')

        try:
            return msg.parsed  # Pydantic validates on parse
        except ValidationError as e:
            if attempt == max_retries - 1:
                raise
            # Add corrective feedback for the next attempt
            messages.append({'role': 'assistant', 'content': msg.content})
            messages.append({'role': 'user', 'content': f'The previous extraction failed validation: {e}\nPlease correct and try again.'})
            print(f'Attempt {attempt+1} failed. Retrying with feedback...')

Business Logic Validation

Business logic validation checks properties that span multiple fields or that depend on external data sources. Pydantic's model_validator runs after all field-level validators and can access the fully-populated model, making it the right place for cross-field checks.

from pydantic import BaseModel, Field, model_validator
from typing import List

class LineItem(BaseModel):
    description: str
    quantity: int = Field(ge=1)
    unit_price: float = Field(ge=0)
    line_total: float

    @model_validator(mode='after')
    def check_line_total(self):
        expected = round(self.quantity * self.unit_price, 2)
        actual = round(self.line_total, 2)
        if abs(expected - actual) > 0.02:  # Allow 2-cent rounding tolerance
            raise ValueError(
                f'Line total {actual} does not match quantity*price={expected}'
            )
        return self

class Invoice(BaseModel):
    line_items: List[LineItem]
    subtotal: float
    tax: float
    total: float

    @model_validator(mode='after')
    def check_invoice_total(self):
        expected_total = round(self.subtotal + self.tax, 2)
        if abs(expected_total - round(self.total, 2)) > 0.02:
            raise ValueError(
                f'Invoice total {self.total} != subtotal+tax ({expected_total})'
            )
        return self

Logging Validation Failures

Every validation failure is a signal about where your pipeline is breaking down. Log each failure with: the input text (or a hash of it for privacy), the extracted output, the specific validation error, and the attempt number. Aggregate these logs to identify systematic patterns — is the model consistently getting one field wrong? Is there a class of documents that causes failures? This data drives targeted prompt improvements.

import logging
from pydantic import ValidationError

logger = logging.getLogger(__name__)

def extract_with_logging(text: str, doc_id: str) -> dict:
    result = None
    for attempt in range(3):
        try:
            result = run_extraction(text)  # Your extraction function
            logger.info('Extraction success', extra={
                'doc_id': doc_id,
                'attempt': attempt + 1
            })
            return result
        except ValidationError as e:
            logger.warning('Validation failure', extra={
                'doc_id': doc_id,
                'attempt': attempt + 1,
                'errors': e.errors(),
                'error_count': len(e.errors())
            })
    # All retries failed
    logger.error('Extraction failed after max retries', extra={'doc_id': doc_id})
    return {'error': 'extraction_failed', 'doc_id': doc_id}

def run_extraction(text):
    pass  # Placeholder for actual extraction logic

Confidence Scores and Thresholds

Add a confidence field to your extraction schema and instruct the model to rate its confidence in each extraction from 0 to 1. Then apply business rules based on confidence: high-confidence extractions go straight to your database, medium-confidence extractions are flagged for spot-checking, and low-confidence extractions go to a human review queue.

This probabilistic approach is far more practical than requiring 100% accuracy from the LLM — you design your pipeline to handle uncertainty gracefully rather than pretending it does not exist.

from pydantic import BaseModel, Field
from typing import Optional

class ExtractedWithConfidence(BaseModel):
    value: Optional[str]
    confidence: float = Field(ge=0.0, le=1.0)
    reason: Optional[str] = None  # Why confidence is low, if below threshold

class DocumentExtraction(BaseModel):
    vendor_name: ExtractedWithConfidence
    invoice_amount: ExtractedWithConfidence
    due_date: ExtractedWithConfidence

def route_by_confidence(extraction: DocumentExtraction, threshold=0.85):
    low_confidence_fields = []
    for field_name, field_val in extraction.model_dump().items():
        if isinstance(field_val, dict) and field_val.get('confidence', 1.0) < threshold:
            low_confidence_fields.append(field_name)

    if not low_confidence_fields:
        return 'auto_approve'
    elif len(low_confidence_fields) > 2:
        return 'human_review'
    else:
        return f'spot_check: {low_confidence_fields}'

Fallback Strategies When Retries Fail

When all retries are exhausted and validation still fails, you need a fallback strategy. Options in order of preference:

Partial result: Return the fields that did validate and mark the failing fields as null
Human review queue: Add the document to a queue for manual review, especially for high-value documents
Lower-fidelity extraction: Fall back to a simpler schema that asks for fewer fields, accepting less structure for robustness
Raw text storage: Store the original text with metadata for later re-processing when you have improved your extraction pipeline

Never silently discard the document. Always log the failure and ensure you can return to it.

Semantic Validation Against External Data

Some validation rules require external data lookups that cannot be done inside Pydantic validators. For example, checking that an extracted company name appears in your CRM, or that an extracted product SKU exists in your inventory. These checks belong in a post-extraction validation step that runs after Pydantic validation passes.

from typing import Optional

# Simulated external data source
KNOWN_VENDORS = {'acme corp', 'techsupplies inc', 'globex corporation'}

def validate_against_crm(extraction: dict) -> dict:
    vendor = extraction.get('vendor', '').lower()
    warnings = []

    if vendor and vendor not in KNOWN_VENDORS:
        warnings.append({
            'field': 'vendor',
            'issue': f'Vendor "{vendor}" not found in CRM',
            'severity': 'warning'
        })
        # Optionally suggest closest match
        # from difflib import get_close_matches
        # matches = get_close_matches(vendor, KNOWN_VENDORS, n=1, cutoff=0.8)
        # if matches: warnings[-1]['suggestion'] = matches[0]

    return {
        'extraction': extraction,
        'warnings': warnings,
        'requires_review': len(warnings) > 0
    }

print('Semantic validation pattern defined')

Testing Your Validation Pipeline

Your validation logic needs its own test suite, separate from the extraction tests. Write unit tests that feed known-bad outputs into your validators and verify the correct errors are raised. Test edge cases: amounts at boundary values, dates in unusual formats, fields with unexpected whitespace, and fields that are numeric strings rather than numbers.

This validation test suite runs without any API calls, making it fast and cheap to run on every code change. It is also the best documentation of your validation rules — the test cases make explicit every format and business constraint your pipeline enforces.

from pydantic import ValidationError

def test_extraction_validation():
    # Test cases: (input_data, should_pass)
    test_cases = [
        ({'vendor': 'ACME', 'amount': 150.00, 'currency': 'USD', 'invoice_date': '2025-01-15'}, True),
        ({'vendor': 'ACME', 'amount': -50.00, 'currency': 'USD', 'invoice_date': '2025-01-15'}, False),  # negative
        ({'vendor': 'ACME', 'amount': 150.00, 'currency': 'EURO', 'invoice_date': '2025-01-15'}, False), # 4-char
        ({'vendor': 'ACME', 'amount': 150.00, 'currency': 'USD', 'invoice_date': 'yesterday'}, False),   # bad date
    ]
    passed = failed = 0
    for data, should_pass in test_cases:
        try:
            # ExtractedInvoice(**data)  # Your Pydantic model
            if should_pass:
                passed += 1
            else:
                print(f'MISSED: Should have failed for {data}')
                failed += 1
        except (ValidationError, ValueError):
            if not should_pass:
                passed += 1
            else:
                print(f'UNEXPECTED FAIL for {data}')
                failed += 1
    print(f'Tests: {passed} passed, {failed} failed')

test_extraction_validation()

Pattern Failure Analysis and Prompt Tuning

After running your extraction pipeline on a sample of real documents and reviewing validation failures, you will likely find that 80% of failures share a small number of root causes. Common culprits: the model consistently misformats dates from a specific locale, misidentifies the tax amount vs. the total, or produces unhyphenated phone numbers when your validator expects hyphens.

For each recurring failure pattern, update your extraction prompt with a specific example and constraint that prevents that error. After updating, rerun your full test set to confirm the fix improved accuracy without regressing other cases. This iterative prompt-then-evaluate cycle is how production extraction pipelines reach 95%+ accuracy on real-world data.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: validation operates at multiple layers — schema, format, business logic, and semantic — each requiring different implementation approaches, retry with corrective feedback provides the model with specific error context to fix its output efficiently, and confidence scores enable probabilistic routing to auto-approve, spot-check, or human-review queues based on extraction certainty. You have now completed the Structured Output and JSON Mode course. Next up we explore vector embeddings — the foundation of every RAG system.

Frequently asked questions

Is the “Validating and Retrying Bad Outputs” lesson free?

Yes — the full text of “Validating and Retrying Bad Outputs” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Validating and Retrying Bad Outputs”?

Implement a validation layer that checks extracted data against business rules, automatically retries with corrective feedback when validation fails, and logs failure patterns. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Validating and Retrying Bad Outputs” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Engineering Academy