从非结构化文本中提取数据
构建信息提取流程，读取电子邮件、收据和文章等原始文本，并返回包含类型、默认值和验证规则的结构化字段。
从非结构化文本中提取数据 是 CoddyKit 上的免费 AI Engineering Academy 课时。 这是第 3 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
The Information Extraction Problem
Organizations are drowning in unstructured text: emails, support tickets, contracts, invoices, news articles, medical notes, and social media posts. Valuable structured data is buried in this text, but extracting it manually is slow, expensive, and error-prone. LLMs with structured outputs change this: they can read any text and populate a predefined schema with the relevant fields, at scale, with reasonable accuracy.
Information extraction (IE) is the process of automatically identifying and pulling structured facts from unstructured text. LLM-based IE dramatically outperforms earlier rule-based or classical NLP approaches because LLMs understand context, synonymy, and implicit information without needing hand-crafted regex patterns for every variation.
Common Extraction Use Cases
Information extraction powers many valuable business applications:
- Invoice processing: Extract vendor, line items, amounts, and due dates from PDF invoices for accounts payable automation
- Contract analysis: Extract parties, effective dates, payment terms, and termination clauses from legal documents
- Resume parsing: Extract skills, experience, education, and contact info from CVs for ATS systems
- Support ticket routing: Extract category, severity, affected product, and customer tier to route tickets automatically
- News monitoring: Extract entities, events, and sentiments from news articles for competitive intelligence
Building an Email Extraction Pipeline
Let us build a practical extraction pipeline that reads customer emails and extracts actionable structured data. The pipeline uses a Pydantic schema to define exactly what we want from each email, then processes emails in batch.
import openai
from pydantic import BaseModel
from typing import List, Optional
from enum import Enum
client = openai.OpenAI()
class Priority(str, Enum):
urgent = 'urgent'
high = 'high'
normal = 'normal'
low = 'low'
class EmailExtraction(BaseModel):
subject_summary: str
sender_intent: str
product_mentioned: Optional[str]
issue_category: str # billing / technical / general / feedback
priority: Priority
action_required: bool
action_description: Optional[str]
customer_sentiment: str # positive / negative / neutral / frustrated
def extract_from_email(email_body: str) -> EmailExtraction:
result = client.beta.chat.completions.parse(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': 'You are an expert at analyzing customer emails and extracting structured information for a support team.'},
{'role': 'user', 'content': f'Analyze this customer email:\n\n{email_body}'}
],
response_format=EmailExtraction
)
return result.choices[0].message.parsedNamed Entity Recognition with LLMs
Named Entity Recognition (NER) is a classic IE task: identifying and classifying named entities (people, organizations, locations, dates, monetary amounts) in text. LLMs dramatically simplify NER because you just describe the entities you want and they extract them without needing a specially trained NER model.
import openai
from pydantic import BaseModel
from typing import List, Optional
client = openai.OpenAI()
class NamedEntity(BaseModel):
text: str # The exact text as it appears
entity_type: str # PERSON / ORG / LOCATION / DATE / MONEY / PRODUCT
normalized: Optional[str] # Standardized form where applicable
class NERResult(BaseModel):
entities: List[NamedEntity]
text = '''
Apple Inc. CEO Tim Cook announced yesterday that the company will invest $1.2 billion
in a new manufacturing facility in Austin, Texas, expected to open in Q3 2026.
'''
result = client.beta.chat.completions.parse(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': 'Extract all named entities from the text. Classify each as PERSON, ORG, LOCATION, DATE, MONEY, or PRODUCT.'},
{'role': 'user', 'content': text}
],
response_format=NERResult
)
for entity in result.choices[0].message.parsed.entities:
print(f'[{entity.entity_type}] {entity.text}')Extracting from Documents at Scale
For production extraction pipelines processing thousands of documents, you need async processing and rate limit handling. A typical pattern uses asyncio with a semaphore to process documents in parallel while respecting the API's rate limits.
import asyncio
import openai
from pydantic import BaseModel
from typing import List, Optional
async_client = openai.AsyncOpenAI()
class InvoiceExtraction(BaseModel):
vendor: str
total_amount: Optional[float]
currency: str
invoice_date: Optional[str]
async def extract_invoice(doc_text: str, semaphore: asyncio.Semaphore) -> InvoiceExtraction:
async with semaphore: # Limit concurrent requests
result = await async_client.beta.chat.completions.parse(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': 'Extract invoice data.'},
{'role': 'user', 'content': doc_text}
],
response_format=InvoiceExtraction
)
return result.choices[0].message.parsed
async def process_invoices(documents: List[str]):
sem = asyncio.Semaphore(5) # Max 5 concurrent requests
tasks = [extract_invoice(doc, sem) for doc in documents]
return await asyncio.gather(*tasks, return_exceptions=True)
# results = asyncio.run(process_invoices(invoice_texts))
print('Async pipeline defined - handles rate limits via semaphore')Handling Implicit and Inferred Information
LLMs can extract not just explicitly stated information but also inferred or implicit information. If a review says 'I have been using this daily for a month and it still works perfectly', the model can infer durability as a positive attribute even though the word 'durability' never appears. This is a major advantage over regex-based extraction which can only find what is explicitly present.
However, this power comes with risk: the model may over-infer and populate fields with guesses rather than facts. For high-stakes extraction (legal, financial, medical), add a confidence field to your schema and instruct the model to rate its certainty, flagging low-confidence extractions for human review.
Extraction Prompt Design
The quality of your extraction depends heavily on prompt design. Key principles for extraction prompts:
- Define ambiguous fields: If 'date' could mean invoice date, due date, or received date, specify exactly which one you want
- Provide examples for unusual formats: 'For price, return the numeric value only, e.g., 29.99 not $29.99'
- Handle normalization: 'Normalize country names to ISO 3166-1 alpha-2 codes'
- Specify extraction source: 'Extract only from the subject line, not the email body'
Think of the extraction prompt as a precise specification for a human data entry operator — every ambiguity you leave in the prompt is a judgment call the model will make inconsistently.
Multi-Pass Extraction for Complex Documents
Some documents are too complex to extract in a single pass because the full schema is large, different sections require different expertise, or the document structure is highly variable. Multi-pass extraction breaks the task into sequential steps: first classify the document type, then extract the appropriate schema for that type.
import openai
from pydantic import BaseModel
from typing import Optional
client = openai.OpenAI()
class DocumentType(BaseModel):
doc_type: str # invoice / contract / resume / report
confidence: float
def classify_document(text: str) -> str:
result = client.beta.chat.completions.parse(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': 'Classify the document type.'},
{'role': 'user', 'content': text[:500]} # Use only the beginning for classification
],
response_format=DocumentType
)
return result.choices[0].message.parsed.doc_type
# Then route to the appropriate extraction schema
EXTRACTION_SCHEMAS = {
'invoice': 'InvoiceSchema', # Replace with actual Pydantic classes
'contract': 'ContractSchema',
'resume': 'ResumeSchema',
}
print('Multi-pass: classify first, then extract with the right schema')Post-Extraction Enrichment
Extracted data often needs enrichment after initial extraction: converting extracted company names to canonical forms by querying a company database, looking up the extracted zip code to fill in city and state, or converting extracted dates to a standard format. This enrichment step should happen in your application code after extraction, not during the LLM call itself.
Keeping extraction and enrichment separate makes the pipeline easier to test and maintain. You can unit-test the enrichment logic independently and swap out the extraction model without changing your enrichment code.
Measuring Extraction Accuracy
For production extraction pipelines, measure accuracy systematically against a labeled test set. Key metrics are:
- Field accuracy: Percentage of fields extracted correctly per document
- Exact match: Field value matches the ground truth exactly
- Normalized match: Field value matches after normalization (e.g., '$1,234.00' == '1234.0')
- False positive rate: How often the model extracts a field that should be null
- False negative rate: How often the model returns null for a field that is present
Run this evaluation whenever you change models, update prompts, or add new document sources to your pipeline. Even small accuracy drops can cascade into significant business impact when processing thousands of documents.
Extraction Logging for Continuous Improvement
Every extraction result in production is a data point that can improve your pipeline. Log every input document, extracted output, and any validation errors to a database. Periodically sample from production logs to identify common failure patterns: certain document formats the model struggles with, fields that are frequently null when they should not be, or unusual values that indicate prompt drift.
This logged data also becomes your future training dataset if you ever want to fine-tune a model specifically for your extraction task, giving you a higher-accuracy, lower-cost alternative to general-purpose LLMs for structured extraction.
Quick Check
Test your understanding of AI Engineering concepts from this lesson.
Lesson Recap
In this lesson you learned: LLMs with Pydantic schemas extract structured data from any unstructured text source reliably, async processing with semaphores enables batch extraction of thousands of documents while respecting rate limits, and multi-pass extraction classifies documents first then applies the appropriate schema for each type. Next up we build validation and auto-retry logic to handle cases where extracted data fails business rules.
常见问题解答
「从非结构化文本中提取数据」课时是免费的吗？
是的 — 「从非结构化文本中提取数据」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。
「从非结构化文本中提取数据」这节课中我会学到什么？
构建信息提取流程，读取电子邮件、收据和文章等原始文本，并返回包含类型、默认值和验证规则的结构化字段。 你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 AI Engineering Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 3 节课，共 4 节。
「从非结构化文本中提取数据」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 AI Engineering Academy 课中编写并运行代码吗？
能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。