设计生产环境架构
选择一个综合项目，绘制包含 RAG 流程、智能体层、缓存、可观测性和 API 的完整架构，然后记录设计决策及其权衡。
设计生产环境架构 是 CoddyKit 上的免费 AI Engineering Academy 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Choosing a Capstone Project
The capstone project ties together everything from the track: RAG, agents, streaming, caching, observability, and security. A good capstone project is meaningfully complex — requiring at least three distinct AI components — yet scoped small enough to ship in days rather than months. Classic examples: a production-grade document Q&A assistant, an autonomous research agent with human-in-the-loop oversight, or an enterprise data extraction pipeline.
Identifying System Components
Start by listing the distinct components your system needs. A production AI engineering project typically includes: an ingestion layer (document loading, chunking, embedding, vector store indexing), a retrieval layer (hybrid search, re-ranking), an agent layer (function calling, tool execution), an API layer (FastAPI backend, streaming endpoints), and an observability layer (tracing, metrics, alerts). Sketch the data flow between them before writing code.
# Component inventory for a Document QA Assistant:
COMPONENTS = [
'document_ingestion', # PDF/Word -> chunks -> embeddings -> pgvector
'hybrid_retriever', # BM25 + dense + RRF
'reranker', # Cohere rerank
'qa_agent', # GPT-4o with RAG + function calling
'semantic_cache', # Redis + embedding similarity
'streaming_api', # FastAPI StreamingResponse
'tracing', # LangSmith or Langfuse
'prompt_injection_filter', # Input sanitization
'eval_pipeline', # Automated quality scoring
]Choosing the Right Stack
Select technology based on your team's familiarity and the system's actual requirements rather than novelty. A reasonable default stack: FastAPI for the API layer, pgvector for vector storage (reuses existing PostgreSQL infrastructure), LangChain LCEL for pipeline composition, Redis for semantic caching and rate-limit state, LangSmith for tracing, and PostgreSQL for user data and evaluation results. Add components only when simpler options do not meet requirements.
# Technology decisions and their rationale
STACK = {
'api': ('FastAPI', 'Async support, OpenAPI docs, streaming easy'),
'vector_store': ('pgvector', 'Already on PostgreSQL, no extra infra'),
'llm_primary': ('gpt-4o', 'Best quality for the use case'),
'llm_fallback': ('claude-3.5', 'Different provider for resilience'),
'cache': ('Redis', 'Sub-ms lookup, TTL support built-in'),
'tracing': ('LangSmith', 'Native LangChain integration'),
'embedding': ('text-embedding-3-small', 'Good quality, low cost'),
'reranker': ('Cohere', 'Best-in-class rerank API'),
}Drawing the Architecture Diagram
Document the system architecture as a data-flow diagram showing each component, the data that flows between them, and the direction of flow. Include both the ingestion path (offline: documents → chunks → embeddings → vector store) and the query path (online: user query → cache check → retrieval → reranking → LLM → streaming response). This diagram is your north star for implementation and helps new team members understand the system instantly.
# Data flow (ASCII art):
#
# INGESTION PATH (offline):
# Documents --> Loader --> Chunker --> Embedder --> pgvector
# |
# BM25 index
#
# QUERY PATH (online):
# User Query
# |-> Semantic Cache (hit: return) -> miss:
# |-> Hybrid Retriever (BM25 + dense)
# |-> Cohere Reranker
# |-> LangChain LCEL Chain
# |-> GPT-4o (streaming) --> FastAPI StreamingResponse
# |-> LangSmith (trace every step)Defining API Contracts Early
Define your API endpoints and their request/response schemas in FastAPI before implementing backend logic. This creates a contract between the API and any frontend consumers and makes parallel development possible. Document every endpoint with OpenAPI descriptions. At minimum, define endpoints for: document ingestion, chat/query, conversation history, evaluation results, and system health.
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI(title='Document QA Assistant', version='1.0')
class QueryRequest(BaseModel):
question: str
conversation_id: str | None = None
max_chunks: int = 5
stream: bool = True
class QueryResponse(BaseModel):
answer: str
sources: list
cached: bool
latency_ms: int
trace_id: str
@app.post('/query', response_model=QueryResponse)
async def query_endpoint(request: QueryRequest):
pass # implementation next lessonEstimating and Budgeting Costs
Estimate your system's monthly API cost before writing a single line of backend code. Count: expected daily active users × queries per user × average tokens per query. For a system with 100 DAU, 10 queries/day, and 3,000 tokens per query at GPT-4o pricing, that is 3 million tokens per day — roughly $45/day or $1,350/month. This estimate tells you whether the system is economically viable and which optimizations (caching, model routing) are worth implementing.
# Cost estimation model
DAU = 100 # daily active users
QPD = 10 # queries per user per day
TOKENS_PER_QUERY = {
'prompt_tokens': 2000, # system + context + question
'completion_tokens': 500
}
PRICE_GPT4O_INPUT = 2.50 / 1_000_000 # per token
PRICE_GPT4O_OUTPUT = 10.00 / 1_000_000 # per token
daily_cost = DAU * QPD * (
TOKENS_PER_QUERY['prompt_tokens'] * PRICE_GPT4O_INPUT +
TOKENS_PER_QUERY['completion_tokens'] * PRICE_GPT4O_OUTPUT
)
print(f'Daily: ${daily_cost:.2f}, Monthly: ${daily_cost * 30:.2f}')Planning the Ingestion Pipeline
Design the ingestion pipeline as an offline batch process that runs on demand or on a schedule. Define what document types you will support (PDF, DOCX, HTML, plain text), the chunking strategy, the embedding model, and the metadata fields to store alongside each vector. Metadata is critical for filtered retrieval — without it, you cannot restrict search to documents from a specific date range, author, or category.
from dataclasses import dataclass
from typing import Optional
@dataclass
class ChunkMetadata:
doc_id: str
source_file: str
page_number: Optional[int]
section_title: Optional[str]
created_at: str
author: Optional[str]
doc_type: str # 'pdf', 'docx', 'html'
# Ingestion config
INGESTION_CONFIG = {
'chunk_size': 800,
'chunk_overlap': 100,
'embedding_model': 'text-embedding-3-small',
'embedding_dimensions': 1536,
'batch_size': 100, # chunks per embedding API call
}Security Architecture Decisions
Make security decisions upfront rather than retrofitting them later. Define: how users authenticate (JWT, API keys), what data can be retrieved per user (row-level security in pgvector queries), how prompt injection is detected, what output scanning is applied, and which actions require multi-factor confirmation. Each decision has performance implications that affect architecture choices throughout the system.
SECURITY_DECISIONS = {
'auth': 'JWT with 24h expiry',
'data_isolation': 'tenant_id column in all vector metadata, filter on every query',
'injection_detection': 'rule-based pre-filter + LLM secondary check for complex inputs',
'output_scanning': 'check for PII, system prompt leakage patterns',
'destructive_actions': 'require confirmation token for delete operations',
'rate_limiting': '20 queries/minute per user, 429 with Retry-After header',
'key_storage': 'AWS Secrets Manager, rotated every 90 days'
}Defining Success Metrics
Define what success looks like for your system before you build it. A set of concrete, measurable success criteria keeps development focused and gives you clear go/no-go criteria for deployment. Include metrics for: quality (eval score on test set), latency (p95 TTFT and total), cost (per-query cost target), and reliability (uptime SLA). Post these on the project README so all contributors share the same target.
SUCCESS_METRICS = {
# Quality
'min_correctness_score': 4.0, # out of 5, LLM-as-judge
'min_retrieval_hit_rate': 0.85, # top-5 chunk contains answer
# Latency
'p95_ttft_ms': 800, # time to first token
'p95_total_latency_ms': 8000, # full response
# Cost
'max_cost_per_query_usd': 0.05, # $0.05 per Q&A
# Reliability
'target_uptime': 0.999, # 99.9%
'cache_hit_rate_target': 0.25, # 25% queries served from cache
}Documenting Architecture Trade-offs
Every architecture decision involves trade-offs. Document them explicitly in an Architecture Decision Record (ADR): what decision was made, what alternatives were considered, and why this option was chosen. For example: 'Chose pgvector over Pinecone because we already run PostgreSQL, reducing operational overhead. Trade-off: maximum scale is limited to ~10M vectors without sharding.' Future team members will thank you for this transparency.
# Architecture Decision Records (ADRs):
#
# ADR-001: Use pgvector for vector storage
# Decision: pgvector in existing PostgreSQL
# Alternatives: Pinecone, Weaviate, Qdrant
# Reason: No new infra, row-level security native, familiar operations
# Trade-offs: Limited to ~5M vectors before performance degrades
#
# ADR-002: GPT-4o as primary model
# Decision: gpt-4o for all user-facing queries
# Alternatives: gpt-4o-mini (cheaper), Claude (alternative)
# Reason: Highest quality for use case, Claude as fallback
# Trade-offs: $0.04/query vs $0.002 for gpt-4o-miniSketching the Deployment Topology
Define how your system will be deployed before writing any infrastructure code. Map each component to a deployment unit: the FastAPI backend as a Docker container, the ingestion pipeline as a separate worker service, pgvector as a managed PostgreSQL instance, Redis as a managed cache. Specify which components are stateless (can be scaled horizontally) versus stateful (require careful scaling strategies). A deployment diagram saves hours of rework later.
# Deployment topology:
#
# Internet -> Load Balancer (AWS ALB)
# |
# FastAPI API (stateless, 2-4 containers, auto-scale)
# |
# +------+------+
# | |
# pgvector Redis Cache
# (AWS RDS Postgres) (Elasticache)
# |
# Ingestion Worker (separate container, manual trigger)
# |
# LangSmith (external SaaS, traces only)
#
# All containers in same VPC, no public access to DB/cacheQuick Check
Test your understanding of production AI system architecture design.
Lesson Recap
In this lesson you learned: component inventory and data-flow diagrams create a shared architectural vision before coding begins, API contracts defined upfront enable parallel development and make requirements explicit, and success metrics defined in advance keep the team aligned on quality, latency, cost, and reliability targets. Next up we implement the core RAG and agent features.
常见问题解答
「设计生产环境架构」课时是免费的吗？
是的 — 「设计生产环境架构」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。
「设计生产环境架构」这节课中我会学到什么？
选择一个综合项目，绘制包含 RAG 流程、智能体层、缓存、可观测性和 API 的完整架构，然后记录设计决策及其权衡。 你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 AI Engineering Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。
「设计生产环境架构」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 AI Engineering Academy 课中编写并运行代码吗？
能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。