0Pricing
AI Engineering Academy · Урок

Извлечение данных из неструктурированного текста

Создайте конвейер извлечения информации, который читает исходный текст, например электронные письма, чеки и статьи, и возвращает структурированные поля с типами, значениями по умолчанию и проверкой.

«Извлечение данных из неструктурированного текста» — бесплатный урок AI Engineering Academy на CoddyKit. Это урок 3 из 4. Ты можешь прочитать весь урок бесплатно ниже — а потом практиковать его прямо в браузере с встроенным редактором кода и ИИ-репетитором 24/7. Это часть пути обучения AI Engineering Academy, и твой прогресс синхронизируется между веб-версией и приложением CoddyKit. Курс AI Engineering Academy содержит 4 уроков всего.

Части этого урока еще не переведены и отображаются на английском.

The Information Extraction Problem

Organizations are drowning in unstructured text: emails, support tickets, contracts, invoices, news articles, medical notes, and social media posts. Valuable structured data is buried in this text, but extracting it manually is slow, expensive, and error-prone. LLMs with structured outputs change this: they can read any text and populate a predefined schema with the relevant fields, at scale, with reasonable accuracy.

Information extraction (IE) is the process of automatically identifying and pulling structured facts from unstructured text. LLM-based IE dramatically outperforms earlier rule-based or classical NLP approaches because LLMs understand context, synonymy, and implicit information without needing hand-crafted regex patterns for every variation.

Common Extraction Use Cases

Information extraction powers many valuable business applications:

  • Invoice processing: Extract vendor, line items, amounts, and due dates from PDF invoices for accounts payable automation
  • Contract analysis: Extract parties, effective dates, payment terms, and termination clauses from legal documents
  • Resume parsing: Extract skills, experience, education, and contact info from CVs for ATS systems
  • Support ticket routing: Extract category, severity, affected product, and customer tier to route tickets automatically
  • News monitoring: Extract entities, events, and sentiments from news articles for competitive intelligence

Building an Email Extraction Pipeline

Let us build a practical extraction pipeline that reads customer emails and extracts actionable structured data. The pipeline uses a Pydantic schema to define exactly what we want from each email, then processes emails in batch.

import openai
from pydantic import BaseModel
from typing import List, Optional
from enum import Enum

client = openai.OpenAI()

class Priority(str, Enum):
    urgent = 'urgent'
    high = 'high'
    normal = 'normal'
    low = 'low'

class EmailExtraction(BaseModel):
    subject_summary: str
    sender_intent: str
    product_mentioned: Optional[str]
    issue_category: str  # billing / technical / general / feedback
    priority: Priority
    action_required: bool
    action_description: Optional[str]
    customer_sentiment: str  # positive / negative / neutral / frustrated

def extract_from_email(email_body: str) -> EmailExtraction:
    result = client.beta.chat.completions.parse(
        model='gpt-4o-mini',
        messages=[
            {'role': 'system', 'content': 'You are an expert at analyzing customer emails and extracting structured information for a support team.'},
            {'role': 'user', 'content': f'Analyze this customer email:\n\n{email_body}'}
        ],
        response_format=EmailExtraction
    )
    return result.choices[0].message.parsed

Named Entity Recognition with LLMs

Named Entity Recognition (NER) is a classic IE task: identifying and classifying named entities (people, organizations, locations, dates, monetary amounts) in text. LLMs dramatically simplify NER because you just describe the entities you want and they extract them without needing a specially trained NER model.

import openai
from pydantic import BaseModel
from typing import List, Optional

client = openai.OpenAI()

class NamedEntity(BaseModel):
    text: str       # The exact text as it appears
    entity_type: str  # PERSON / ORG / LOCATION / DATE / MONEY / PRODUCT
    normalized: Optional[str]  # Standardized form where applicable

class NERResult(BaseModel):
    entities: List[NamedEntity]

text = '''
Apple Inc. CEO Tim Cook announced yesterday that the company will invest $1.2 billion
in a new manufacturing facility in Austin, Texas, expected to open in Q3 2026.
'''

result = client.beta.chat.completions.parse(
    model='gpt-4o-mini',
    messages=[
        {'role': 'system', 'content': 'Extract all named entities from the text. Classify each as PERSON, ORG, LOCATION, DATE, MONEY, or PRODUCT.'},
        {'role': 'user', 'content': text}
    ],
    response_format=NERResult
)
for entity in result.choices[0].message.parsed.entities:
    print(f'[{entity.entity_type}] {entity.text}')

Extracting from Documents at Scale

For production extraction pipelines processing thousands of documents, you need async processing and rate limit handling. A typical pattern uses asyncio with a semaphore to process documents in parallel while respecting the API's rate limits.

import asyncio
import openai
from pydantic import BaseModel
from typing import List, Optional

async_client = openai.AsyncOpenAI()

class InvoiceExtraction(BaseModel):
    vendor: str
    total_amount: Optional[float]
    currency: str
    invoice_date: Optional[str]

async def extract_invoice(doc_text: str, semaphore: asyncio.Semaphore) -> InvoiceExtraction:
    async with semaphore:  # Limit concurrent requests
        result = await async_client.beta.chat.completions.parse(
            model='gpt-4o-mini',
            messages=[
                {'role': 'system', 'content': 'Extract invoice data.'},
                {'role': 'user', 'content': doc_text}
            ],
            response_format=InvoiceExtraction
        )
        return result.choices[0].message.parsed

async def process_invoices(documents: List[str]):
    sem = asyncio.Semaphore(5)  # Max 5 concurrent requests
    tasks = [extract_invoice(doc, sem) for doc in documents]
    return await asyncio.gather(*tasks, return_exceptions=True)

# results = asyncio.run(process_invoices(invoice_texts))
print('Async pipeline defined - handles rate limits via semaphore')

Handling Implicit and Inferred Information

LLMs can extract not just explicitly stated information but also inferred or implicit information. If a review says 'I have been using this daily for a month and it still works perfectly', the model can infer durability as a positive attribute even though the word 'durability' never appears. This is a major advantage over regex-based extraction which can only find what is explicitly present.

However, this power comes with risk: the model may over-infer and populate fields with guesses rather than facts. For high-stakes extraction (legal, financial, medical), add a confidence field to your schema and instruct the model to rate its certainty, flagging low-confidence extractions for human review.

Extraction Prompt Design

The quality of your extraction depends heavily on prompt design. Key principles for extraction prompts:

  • Define ambiguous fields: If 'date' could mean invoice date, due date, or received date, specify exactly which one you want
  • Provide examples for unusual formats: 'For price, return the numeric value only, e.g., 29.99 not $29.99'
  • Handle normalization: 'Normalize country names to ISO 3166-1 alpha-2 codes'
  • Specify extraction source: 'Extract only from the subject line, not the email body'

Think of the extraction prompt as a precise specification for a human data entry operator — every ambiguity you leave in the prompt is a judgment call the model will make inconsistently.

Multi-Pass Extraction for Complex Documents

Some documents are too complex to extract in a single pass because the full schema is large, different sections require different expertise, or the document structure is highly variable. Multi-pass extraction breaks the task into sequential steps: first classify the document type, then extract the appropriate schema for that type.

import openai
from pydantic import BaseModel
from typing import Optional

client = openai.OpenAI()

class DocumentType(BaseModel):
    doc_type: str  # invoice / contract / resume / report
    confidence: float

def classify_document(text: str) -> str:
    result = client.beta.chat.completions.parse(
        model='gpt-4o-mini',
        messages=[
            {'role': 'system', 'content': 'Classify the document type.'},
            {'role': 'user', 'content': text[:500]}  # Use only the beginning for classification
        ],
        response_format=DocumentType
    )
    return result.choices[0].message.parsed.doc_type

# Then route to the appropriate extraction schema
EXTRACTION_SCHEMAS = {
    'invoice': 'InvoiceSchema',  # Replace with actual Pydantic classes
    'contract': 'ContractSchema',
    'resume': 'ResumeSchema',
}

print('Multi-pass: classify first, then extract with the right schema')

Post-Extraction Enrichment

Extracted data often needs enrichment after initial extraction: converting extracted company names to canonical forms by querying a company database, looking up the extracted zip code to fill in city and state, or converting extracted dates to a standard format. This enrichment step should happen in your application code after extraction, not during the LLM call itself.

Keeping extraction and enrichment separate makes the pipeline easier to test and maintain. You can unit-test the enrichment logic independently and swap out the extraction model without changing your enrichment code.

Measuring Extraction Accuracy

For production extraction pipelines, measure accuracy systematically against a labeled test set. Key metrics are:

  • Field accuracy: Percentage of fields extracted correctly per document
  • Exact match: Field value matches the ground truth exactly
  • Normalized match: Field value matches after normalization (e.g., '$1,234.00' == '1234.0')
  • False positive rate: How often the model extracts a field that should be null
  • False negative rate: How often the model returns null for a field that is present

Run this evaluation whenever you change models, update prompts, or add new document sources to your pipeline. Even small accuracy drops can cascade into significant business impact when processing thousands of documents.

Extraction Logging for Continuous Improvement

Every extraction result in production is a data point that can improve your pipeline. Log every input document, extracted output, and any validation errors to a database. Periodically sample from production logs to identify common failure patterns: certain document formats the model struggles with, fields that are frequently null when they should not be, or unusual values that indicate prompt drift.

This logged data also becomes your future training dataset if you ever want to fine-tune a model specifically for your extraction task, giving you a higher-accuracy, lower-cost alternative to general-purpose LLMs for structured extraction.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: LLMs with Pydantic schemas extract structured data from any unstructured text source reliably, async processing with semaphores enables batch extraction of thousands of documents while respecting rate limits, and multi-pass extraction classifies documents first then applies the appropriate schema for each type. Next up we build validation and auto-retry logic to handle cases where extracted data fails business rules.

Часто задаваемые вопросы

Урок «Извлечение данных из неструктурированного текста» бесплатный?

Да — полный текст урока «Извлечение данных из неструктурированного текста» бесплатно доступен здесь в веб-версии. Чтобы практиковать его интерактивно (встроенный редактор кода и ИИ-репетитор 24/7) и разблокировать остальной курс AI Engineering Academy, подпишись на CoddyKit PRO. Курс AI Engineering Academy содержит 4 уроков всего.

Чему я научусь в уроке «Извлечение данных из неструктурированного текста»?

Создайте конвейер извлечения информации, который читает исходный текст, например электронные письма, чеки и статьи, и возвращает структурированные поля с типами, значениями по умолчанию и проверкой. Ты практикуешь AI Engineering Academy с помощью реального кода, который запускаешь прямо в браузере, и ИИ-репетитор 24/7 отвечает на твои вопросы во время урока.

Нужен ли мне опыт, чтобы начать AI Engineering Academy?

Предыдущий опыт не требуется. AI Engineering Academy на CoddyKit структурирован для всех уровней — от новичков до продвинутых, поэтому ты можешь начать отсюда или с самого начала и учиться в своем темпе. Это урок 3 из 4.

Сколько времени занимает урок «Извлечение данных из неструктурированного текста»?

Большинство уроков CoddyKit занимают около 5–10 минут. Каждый из них компактный и интерактивный, поэтому ты постоянно делаешь прогресс и продолжаешь с того же места в веб-версии и приложении.

Можно ли писать и запускать код в этом уроке AI Engineering Academy?

Да. Каждый урок AI Engineering Academy включает встроенный редактор кода, поэтому ты пишешь и запускаешь реальный код прямо в браузере и получаешь моментальную обратную связь от AI — локальная установка не требуется.

Все уроки этого курса

  1. Режим JSON и response_format
  2. Структурированный вывод с Pydantic
  3. Извлечение данных из неструктурированного текста
  4. Проверка и повторная обработка некорректного вывода
← Назад к AI Engineering Academy