AI Engineering Academy · Ders

Kısmi ve Eksik Verileri İşleme

Kısmi ve Eksik Verileri İşleme, CoddyKit'te ücretsiz bir AI Engineering Academy dersidir. Bu, 4 dersinin 2. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, AI Engineering Academy öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. AI Engineering Academy kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

The Reality of Incomplete Documents

Real-world documents rarely contain every field your schema expects. An invoice might be missing a PO number, a resume might omit dates, and a news article might not mention a location. Designing your extraction schema to handle partial and missing data gracefully is as important as extracting what is present.

Optional Fields in Pydantic

Mark fields that might not appear in every document as Optional[type] and give them a None default. Pydantic v2 treats these fields as nullable, and the model is instructed not to hallucinate values when information is absent. Always prefer None over an empty string for missing data — it is easier to filter downstream.

from pydantic import BaseModel, Field
from typing import Optional

class JobPosting(BaseModel):
    title: str
    company: str
    salary_min: Optional[float] = Field(None, description='Minimum salary if stated')
    salary_max: Optional[float] = Field(None, description='Maximum salary if stated')
    remote: Optional[bool] = Field(None, description='True if remote, False if on-site, None if unspecified')

Adding Confidence Scores

Ask the model to rate its own confidence in each extracted field by adding a confidence score alongside the value. Wrap both in a generic FieldExtract helper. Low-confidence extractions can be routed to human reviewers rather than passed directly to downstream systems, reducing the risk of silently bad data.

from pydantic import BaseModel
from typing import Optional

class Confident(BaseModel):
    value: Optional[str]
    confidence: float  # 0.0 to 1.0

class ContractExtract(BaseModel):
    party_a: Confident
    party_b: Confident
    effective_date: Confident
    termination_clause: Confident

Sentinel Values vs. None

Sometimes the absence of data is itself meaningful. Use Python's Optional plus a Literal enum to distinguish between not mentioned, explicitly stated as none, and unknown. This three-way distinction prevents downstream code from treating an explicitly stated absence the same as a missing mention, which can cause subtle business logic bugs.

from pydantic import BaseModel
from typing import Optional, Literal

class Discount(BaseModel):
    # 'none' = explicitly no discount; None = not mentioned
    discount_type: Optional[Literal['percentage', 'fixed', 'none']] = None
    discount_value: Optional[float] = None

Fallback Extraction Strategies

When a primary extraction call returns too many None fields, try a targeted follow-up prompt that focuses specifically on the missing information. Send just the relevant paragraph alongside the partially extracted model and ask the model to fill only the empty fields. This two-pass approach significantly improves recall on ambiguous documents.

def fill_missing(partial: JobPosting, raw_text: str) -> JobPosting:
    missing = [k for k, v in partial.model_dump().items() if v is None]
    if not missing:
        return partial
    prompt = f'From this text, extract ONLY these fields: {missing}.\n\n{raw_text}'
    supplement = client.chat.completions.create(
        model='gpt-4o-mini',
        response_model=JobPosting,
        messages=[{'role': 'user', 'content': prompt}]
    )
    merged = partial.model_dump()
    for field in missing:
        if getattr(supplement, field) is not None:
            merged[field] = getattr(supplement, field)
    return JobPosting(**merged)

Using Default Values and Factories

For fields that have a sensible default when missing, use Pydantic's default or default_factory. For example, a list of tags should default to an empty list rather than None so downstream code can always iterate over it. Reserve None for fields where the absence must be flagged and handled explicitly.

from pydantic import BaseModel, Field
from typing import List, Optional

class Article(BaseModel):
    title: str
    author: Optional[str] = None
    tags: List[str] = Field(default_factory=list)
    word_count: Optional[int] = None
    published_date: Optional[str] = None

Routing Low-Confidence Extractions

Build a review queue for extractions that have confidence below a threshold. Store low-confidence results in a separate database table with a needs_review flag, expose them in an internal review UI, and allow human annotators to correct them. Feed the corrections back as few-shot examples to improve future extractions.

CONFIDENCE_THRESHOLD = 0.75

def process_extraction(result: ContractExtract, doc_id: str):
    needs_review = any(
        field.confidence < CONFIDENCE_THRESHOLD
        for field in [result.party_a, result.party_b, result.effective_date]
    )
    if needs_review:
        queue_for_human_review(doc_id, result)
    else:
        store_in_production_table(doc_id, result)

Handling Ambiguous Spans of Text

Some fields can be extracted in multiple valid ways from the same text. For example, a date written as 'next Monday' is ambiguous without a reference date. Use a raw_span field to capture the exact text the model used, alongside the normalized value. This preserves the original evidence and makes debugging extractions much easier.

from pydantic import BaseModel
from typing import Optional

class DateField(BaseModel):
    raw_span: Optional[str] = None    # exact text from document
    iso_date: Optional[str] = None    # normalized YYYY-MM-DD
    confidence: float = 1.0

class Contract(BaseModel):
    effective_date: DateField
    expiration_date: DateField

Logging Missing Field Patterns

Track which fields are most frequently None across your document corpus. High missing rates on a required field suggest either the field is genuinely absent in most documents, or your schema description is confusing the model. Logging missing patterns per document type helps you triage schema improvements that will have the biggest impact on data quality.

from collections import Counter

missing_counter = Counter()

def log_missing(result):
    for field, value in result.model_dump().items():
        if value is None:
            missing_counter[field] += 1

# After processing 1000 documents:
for field, count in missing_counter.most_common(5):
    print(f'{field}: {count} missing ({100*count//1000}%)')

Combining Extraction Across Multiple Passes

For complex documents like annual reports or lengthy contracts, a multi-pass extraction strategy works best. In the first pass, extract high-confidence fields that are always present. In subsequent passes, focus on specific sections or paragraphs to extract the harder fields. Merge all passes into a single final record, letting later passes override earlier None values.

def multi_pass_extract(pages: list) -> Invoice:
    # Pass 1: header info from first page
    header = extract_header(pages[0])
    # Pass 2: line items from middle pages
    items = []
    for page in pages[1:-1]:
        items.extend(extract_line_items(page))
    # Pass 3: totals from last page
    totals = extract_totals(pages[-1])
    return Invoice(
        vendor=header.vendor,
        invoice_number=header.invoice_number,
        line_items=items,
        total_amount=totals.total_amount
    )

Schema Design Best Practices

Well-designed schemas reduce missing data naturally. Use narrow, specific field descriptions so the model knows exactly what to look for. Avoid combining two concepts in one field. Add examples in the Field description to guide extraction. A schema that makes the model's job easy will have far fewer missing fields than one that relies on vague labels.

from pydantic import BaseModel, Field

class Address(BaseModel):
    street: str = Field(description='Street number and name, e.g. 123 Main St')
    city: str = Field(description='City name only, no state')
    state: str = Field(description='Two-letter US state code, e.g. CA')
    zip_code: str = Field(description='5-digit ZIP code, e.g. 94105')
    country: str = Field(default='US', description='ISO 3166-1 alpha-2 country code')

Quick Check

Test your understanding of handling partial and missing data in extraction pipelines.

Lesson Recap

In this lesson you learned: Optional fields with None defaults prevent hallucination of missing data, confidence scores and review queues create a safety net for uncertain extractions, and multi-pass extraction improves recall on complex documents by focusing each pass on specific sections. Next up we scale extraction with async processing and queues.

Sıkça Sorulan Sorular

“Kısmi ve Eksik Verileri İşleme” dersi ücretsiz mi?

Evet — “Kısmi ve Eksik Verileri İşleme” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve AI Engineering Academy kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. AI Engineering Academy kursu toplamda 4 dersten oluşur.

“Kısmi ve Eksik Verileri İşleme” dersinde ne öğreneceğim?

Optional alanları ve güven puanlarını içeren şemalar tasarlayın, belirsiz belgeler için geri dönüş çıkarma stratejileri uygulayın ve düşük güvenli çıkarmaları insan incelemesi için günlük kaydına alı… AI Engineering Academy ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

AI Engineering Academy öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te AI Engineering Academy, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 2. dersidir.

“Kısmi ve Eksik Verileri İşleme” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu AI Engineering Academy dersinde kod yazıp çalıştırabilir miyim?

Evet. Her AI Engineering Academy dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

Instructor: Pydantic ile Tür Bilgili Çıkarma
Kısmi ve Eksik Verileri İşleme
Eşzamansız İşleme ve Kuyruklarla Toplu İşleme
Şema Gelişimi ve Geriye Dönük Uyumluluk

← AI Engineering Academy Sayfasına Dön