AI Engineering Academy · Урок

Обработка частичных и отсутствующих данных

«Обработка частичных и отсутствующих данных» — бесплатный урок AI Engineering Academy на CoddyKit. Это урок 2 из 4. Ты можешь прочитать весь урок бесплатно ниже — а потом практиковать его прямо в браузере с встроенным редактором кода и ИИ-репетитором 24/7. Это часть пути обучения AI Engineering Academy, и твой прогресс синхронизируется между веб-версией и приложением CoddyKit. Курс AI Engineering Academy содержит 4 уроков всего.

Части этого урока еще не переведены и отображаются на английском.

The Reality of Incomplete Documents

Real-world documents rarely contain every field your schema expects. An invoice might be missing a PO number, a resume might omit dates, and a news article might not mention a location. Designing your extraction schema to handle partial and missing data gracefully is as important as extracting what is present.

Optional Fields in Pydantic

Mark fields that might not appear in every document as Optional[type] and give them a None default. Pydantic v2 treats these fields as nullable, and the model is instructed not to hallucinate values when information is absent. Always prefer None over an empty string for missing data — it is easier to filter downstream.

from pydantic import BaseModel, Field
from typing import Optional

class JobPosting(BaseModel):
    title: str
    company: str
    salary_min: Optional[float] = Field(None, description='Minimum salary if stated')
    salary_max: Optional[float] = Field(None, description='Maximum salary if stated')
    remote: Optional[bool] = Field(None, description='True if remote, False if on-site, None if unspecified')

Adding Confidence Scores

Ask the model to rate its own confidence in each extracted field by adding a confidence score alongside the value. Wrap both in a generic FieldExtract helper. Low-confidence extractions can be routed to human reviewers rather than passed directly to downstream systems, reducing the risk of silently bad data.

from pydantic import BaseModel
from typing import Optional

class Confident(BaseModel):
    value: Optional[str]
    confidence: float  # 0.0 to 1.0

class ContractExtract(BaseModel):
    party_a: Confident
    party_b: Confident
    effective_date: Confident
    termination_clause: Confident

Sentinel Values vs. None

Sometimes the absence of data is itself meaningful. Use Python's Optional plus a Literal enum to distinguish between not mentioned, explicitly stated as none, and unknown. This three-way distinction prevents downstream code from treating an explicitly stated absence the same as a missing mention, which can cause subtle business logic bugs.

from pydantic import BaseModel
from typing import Optional, Literal

class Discount(BaseModel):
    # 'none' = explicitly no discount; None = not mentioned
    discount_type: Optional[Literal['percentage', 'fixed', 'none']] = None
    discount_value: Optional[float] = None

Fallback Extraction Strategies

When a primary extraction call returns too many None fields, try a targeted follow-up prompt that focuses specifically on the missing information. Send just the relevant paragraph alongside the partially extracted model and ask the model to fill only the empty fields. This two-pass approach significantly improves recall on ambiguous documents.

def fill_missing(partial: JobPosting, raw_text: str) -> JobPosting:
    missing = [k for k, v in partial.model_dump().items() if v is None]
    if not missing:
        return partial
    prompt = f'From this text, extract ONLY these fields: {missing}.\n\n{raw_text}'
    supplement = client.chat.completions.create(
        model='gpt-4o-mini',
        response_model=JobPosting,
        messages=[{'role': 'user', 'content': prompt}]
    )
    merged = partial.model_dump()
    for field in missing:
        if getattr(supplement, field) is not None:
            merged[field] = getattr(supplement, field)
    return JobPosting(**merged)

Using Default Values and Factories

For fields that have a sensible default when missing, use Pydantic's default or default_factory. For example, a list of tags should default to an empty list rather than None so downstream code can always iterate over it. Reserve None for fields where the absence must be flagged and handled explicitly.

from pydantic import BaseModel, Field
from typing import List, Optional

class Article(BaseModel):
    title: str
    author: Optional[str] = None
    tags: List[str] = Field(default_factory=list)
    word_count: Optional[int] = None
    published_date: Optional[str] = None

Routing Low-Confidence Extractions

Build a review queue for extractions that have confidence below a threshold. Store low-confidence results in a separate database table with a needs_review flag, expose them in an internal review UI, and allow human annotators to correct them. Feed the corrections back as few-shot examples to improve future extractions.

CONFIDENCE_THRESHOLD = 0.75

def process_extraction(result: ContractExtract, doc_id: str):
    needs_review = any(
        field.confidence < CONFIDENCE_THRESHOLD
        for field in [result.party_a, result.party_b, result.effective_date]
    )
    if needs_review:
        queue_for_human_review(doc_id, result)
    else:
        store_in_production_table(doc_id, result)

Handling Ambiguous Spans of Text

Some fields can be extracted in multiple valid ways from the same text. For example, a date written as 'next Monday' is ambiguous without a reference date. Use a raw_span field to capture the exact text the model used, alongside the normalized value. This preserves the original evidence and makes debugging extractions much easier.

from pydantic import BaseModel
from typing import Optional

class DateField(BaseModel):
    raw_span: Optional[str] = None    # exact text from document
    iso_date: Optional[str] = None    # normalized YYYY-MM-DD
    confidence: float = 1.0

class Contract(BaseModel):
    effective_date: DateField
    expiration_date: DateField

Logging Missing Field Patterns

Track which fields are most frequently None across your document corpus. High missing rates on a required field suggest either the field is genuinely absent in most documents, or your schema description is confusing the model. Logging missing patterns per document type helps you triage schema improvements that will have the biggest impact on data quality.

from collections import Counter

missing_counter = Counter()

def log_missing(result):
    for field, value in result.model_dump().items():
        if value is None:
            missing_counter[field] += 1

# After processing 1000 documents:
for field, count in missing_counter.most_common(5):
    print(f'{field}: {count} missing ({100*count//1000}%)')

Combining Extraction Across Multiple Passes

For complex documents like annual reports or lengthy contracts, a multi-pass extraction strategy works best. In the first pass, extract high-confidence fields that are always present. In subsequent passes, focus on specific sections or paragraphs to extract the harder fields. Merge all passes into a single final record, letting later passes override earlier None values.

def multi_pass_extract(pages: list) -> Invoice:
    # Pass 1: header info from first page
    header = extract_header(pages[0])
    # Pass 2: line items from middle pages
    items = []
    for page in pages[1:-1]:
        items.extend(extract_line_items(page))
    # Pass 3: totals from last page
    totals = extract_totals(pages[-1])
    return Invoice(
        vendor=header.vendor,
        invoice_number=header.invoice_number,
        line_items=items,
        total_amount=totals.total_amount
    )

Schema Design Best Practices

Well-designed schemas reduce missing data naturally. Use narrow, specific field descriptions so the model knows exactly what to look for. Avoid combining two concepts in one field. Add examples in the Field description to guide extraction. A schema that makes the model's job easy will have far fewer missing fields than one that relies on vague labels.

from pydantic import BaseModel, Field

class Address(BaseModel):
    street: str = Field(description='Street number and name, e.g. 123 Main St')
    city: str = Field(description='City name only, no state')
    state: str = Field(description='Two-letter US state code, e.g. CA')
    zip_code: str = Field(description='5-digit ZIP code, e.g. 94105')
    country: str = Field(default='US', description='ISO 3166-1 alpha-2 country code')

Quick Check

Test your understanding of handling partial and missing data in extraction pipelines.

Lesson Recap

In this lesson you learned: Optional fields with None defaults prevent hallucination of missing data, confidence scores and review queues create a safety net for uncertain extractions, and multi-pass extraction improves recall on complex documents by focusing each pass on specific sections. Next up we scale extraction with async processing and queues.

Часто задаваемые вопросы

Урок «Обработка частичных и отсутствующих данных» бесплатный?

Да — полный текст урока «Обработка частичных и отсутствующих данных» бесплатно доступен здесь в веб-версии. Чтобы практиковать его интерактивно (встроенный редактор кода и ИИ-репетитор 24/7) и разблокировать остальной курс AI Engineering Academy, подпишись на CoddyKit PRO. Курс AI Engineering Academy содержит 4 уроков всего.

Чему я научусь в уроке «Обработка частичных и отсутствующих данных»?

Проектируйте схемы с полями Optional и оценками уверенности, реализуйте резервные стратегии извлечения для неоднозначных документов и записывайте извлечения с низкой уверенностью для проверки человек… Ты практикуешь AI Engineering Academy с помощью реального кода, который запускаешь прямо в браузере, и ИИ-репетитор 24/7 отвечает на твои вопросы во время урока.

Нужен ли мне опыт, чтобы начать AI Engineering Academy?

Предыдущий опыт не требуется. AI Engineering Academy на CoddyKit структурирован для всех уровней — от новичков до продвинутых, поэтому ты можешь начать отсюда или с самого начала и учиться в своем темпе. Это урок 2 из 4.

Сколько времени занимает урок «Обработка частичных и отсутствующих данных»?

Большинство уроков CoddyKit занимают около 5–10 минут. Каждый из них компактный и интерактивный, поэтому ты постоянно делаешь прогресс и продолжаешь с того же места в веб-версии и приложении.

Можно ли писать и запускать код в этом уроке AI Engineering Academy?

Да. Каждый урок AI Engineering Academy включает встроенный редактор кода, поэтому ты пишешь и запускаешь реальный код прямо в браузере и получаешь моментальную обратную связь от AI — локальная установка не требуется.

Все уроки этого курса

Instructor: типизированное извлечение с Pydantic
Обработка частичных и отсутствующих данных
Пакетная обработка с асинхронностью и очередями
Эволюция схем и обратная совместимость

← Назад к AI Engineering Academy