AI Engineering Academy · Leçon

Gérer les données partielles et manquantes

Gérer les données partielles et manquantes est une leçon AI Engineering Academy gratuite sur CoddyKit. Ceci est la leçon 2 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage AI Engineering Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours AI Engineering Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

The Reality of Incomplete Documents

Real-world documents rarely contain every field your schema expects. An invoice might be missing a PO number, a resume might omit dates, and a news article might not mention a location. Designing your extraction schema to handle partial and missing data gracefully is as important as extracting what is present.

Optional Fields in Pydantic

Mark fields that might not appear in every document as Optional[type] and give them a None default. Pydantic v2 treats these fields as nullable, and the model is instructed not to hallucinate values when information is absent. Always prefer None over an empty string for missing data — it is easier to filter downstream.

from pydantic import BaseModel, Field
from typing import Optional

class JobPosting(BaseModel):
    title: str
    company: str
    salary_min: Optional[float] = Field(None, description='Minimum salary if stated')
    salary_max: Optional[float] = Field(None, description='Maximum salary if stated')
    remote: Optional[bool] = Field(None, description='True if remote, False if on-site, None if unspecified')

Adding Confidence Scores

Ask the model to rate its own confidence in each extracted field by adding a confidence score alongside the value. Wrap both in a generic FieldExtract helper. Low-confidence extractions can be routed to human reviewers rather than passed directly to downstream systems, reducing the risk of silently bad data.

from pydantic import BaseModel
from typing import Optional

class Confident(BaseModel):
    value: Optional[str]
    confidence: float  # 0.0 to 1.0

class ContractExtract(BaseModel):
    party_a: Confident
    party_b: Confident
    effective_date: Confident
    termination_clause: Confident

Sentinel Values vs. None

Sometimes the absence of data is itself meaningful. Use Python's Optional plus a Literal enum to distinguish between not mentioned, explicitly stated as none, and unknown. This three-way distinction prevents downstream code from treating an explicitly stated absence the same as a missing mention, which can cause subtle business logic bugs.

from pydantic import BaseModel
from typing import Optional, Literal

class Discount(BaseModel):
    # 'none' = explicitly no discount; None = not mentioned
    discount_type: Optional[Literal['percentage', 'fixed', 'none']] = None
    discount_value: Optional[float] = None

Fallback Extraction Strategies

When a primary extraction call returns too many None fields, try a targeted follow-up prompt that focuses specifically on the missing information. Send just the relevant paragraph alongside the partially extracted model and ask the model to fill only the empty fields. This two-pass approach significantly improves recall on ambiguous documents.

def fill_missing(partial: JobPosting, raw_text: str) -> JobPosting:
    missing = [k for k, v in partial.model_dump().items() if v is None]
    if not missing:
        return partial
    prompt = f'From this text, extract ONLY these fields: {missing}.\n\n{raw_text}'
    supplement = client.chat.completions.create(
        model='gpt-4o-mini',
        response_model=JobPosting,
        messages=[{'role': 'user', 'content': prompt}]
    )
    merged = partial.model_dump()
    for field in missing:
        if getattr(supplement, field) is not None:
            merged[field] = getattr(supplement, field)
    return JobPosting(**merged)

Using Default Values and Factories

For fields that have a sensible default when missing, use Pydantic's default or default_factory. For example, a list of tags should default to an empty list rather than None so downstream code can always iterate over it. Reserve None for fields where the absence must be flagged and handled explicitly.

from pydantic import BaseModel, Field
from typing import List, Optional

class Article(BaseModel):
    title: str
    author: Optional[str] = None
    tags: List[str] = Field(default_factory=list)
    word_count: Optional[int] = None
    published_date: Optional[str] = None

Routing Low-Confidence Extractions

Build a review queue for extractions that have confidence below a threshold. Store low-confidence results in a separate database table with a needs_review flag, expose them in an internal review UI, and allow human annotators to correct them. Feed the corrections back as few-shot examples to improve future extractions.

CONFIDENCE_THRESHOLD = 0.75

def process_extraction(result: ContractExtract, doc_id: str):
    needs_review = any(
        field.confidence < CONFIDENCE_THRESHOLD
        for field in [result.party_a, result.party_b, result.effective_date]
    )
    if needs_review:
        queue_for_human_review(doc_id, result)
    else:
        store_in_production_table(doc_id, result)

Handling Ambiguous Spans of Text

Some fields can be extracted in multiple valid ways from the same text. For example, a date written as 'next Monday' is ambiguous without a reference date. Use a raw_span field to capture the exact text the model used, alongside the normalized value. This preserves the original evidence and makes debugging extractions much easier.

from pydantic import BaseModel
from typing import Optional

class DateField(BaseModel):
    raw_span: Optional[str] = None    # exact text from document
    iso_date: Optional[str] = None    # normalized YYYY-MM-DD
    confidence: float = 1.0

class Contract(BaseModel):
    effective_date: DateField
    expiration_date: DateField

Logging Missing Field Patterns

Track which fields are most frequently None across your document corpus. High missing rates on a required field suggest either the field is genuinely absent in most documents, or your schema description is confusing the model. Logging missing patterns per document type helps you triage schema improvements that will have the biggest impact on data quality.

from collections import Counter

missing_counter = Counter()

def log_missing(result):
    for field, value in result.model_dump().items():
        if value is None:
            missing_counter[field] += 1

# After processing 1000 documents:
for field, count in missing_counter.most_common(5):
    print(f'{field}: {count} missing ({100*count//1000}%)')

Combining Extraction Across Multiple Passes

For complex documents like annual reports or lengthy contracts, a multi-pass extraction strategy works best. In the first pass, extract high-confidence fields that are always present. In subsequent passes, focus on specific sections or paragraphs to extract the harder fields. Merge all passes into a single final record, letting later passes override earlier None values.

def multi_pass_extract(pages: list) -> Invoice:
    # Pass 1: header info from first page
    header = extract_header(pages[0])
    # Pass 2: line items from middle pages
    items = []
    for page in pages[1:-1]:
        items.extend(extract_line_items(page))
    # Pass 3: totals from last page
    totals = extract_totals(pages[-1])
    return Invoice(
        vendor=header.vendor,
        invoice_number=header.invoice_number,
        line_items=items,
        total_amount=totals.total_amount
    )

Schema Design Best Practices

Well-designed schemas reduce missing data naturally. Use narrow, specific field descriptions so the model knows exactly what to look for. Avoid combining two concepts in one field. Add examples in the Field description to guide extraction. A schema that makes the model's job easy will have far fewer missing fields than one that relies on vague labels.

from pydantic import BaseModel, Field

class Address(BaseModel):
    street: str = Field(description='Street number and name, e.g. 123 Main St')
    city: str = Field(description='City name only, no state')
    state: str = Field(description='Two-letter US state code, e.g. CA')
    zip_code: str = Field(description='5-digit ZIP code, e.g. 94105')
    country: str = Field(default='US', description='ISO 3166-1 alpha-2 country code')

Quick Check

Test your understanding of handling partial and missing data in extraction pipelines.

Lesson Recap

In this lesson you learned: Optional fields with None defaults prevent hallucination of missing data, confidence scores and review queues create a safety net for uncertain extractions, and multi-pass extraction improves recall on complex documents by focusing each pass on specific sections. Next up we scale extraction with async processing and queues.

Questions Fréquemment Posées

La leçon « Gérer les données partielles et manquantes » est-elle gratuite ?

Oui — le texte complet de « Gérer les données partielles et manquantes » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours AI Engineering Academy, passe à CoddyKit PRO. Le cours AI Engineering Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Gérer les données partielles et manquantes » ?

Concevez des schémas avec des champs Optional et des scores de confiance, implémentez des stratégies d’extraction de secours pour les documents ambigus et consignez les extractions peu fiables pour u… Tu pratiques AI Engineering Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer AI Engineering Academy ?

Aucune expérience préalable n'est requise. AI Engineering Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 2 sur 4.

Combien de temps prend la leçon « Gérer les données partielles et manquantes » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon AI Engineering Academy ?

Oui. Chaque leçon AI Engineering Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

Instructor : extraction typée avec Pydantic
Gérer les données partielles et manquantes
Traitement par lots avec l’asynchronisme et des files d’attente
Évolution des schémas et compatibilité descendante

← Retour à AI Engineering Academy