评估、部署与复盘
运行完整的评估框架，包括检索指标、LLM-as-judge 质量评分和负载测试；将应用部署到云服务商，并撰写复盘报告，记录经验教训。
评估、部署与复盘 是 CoddyKit 上的免费 AI Engineering Academy 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
The Final Mile: Evaluation Before Shipping
A system is not ready to ship until it has been evaluated end-to-end against real-world conditions. The final evaluation phase combines all evaluation techniques learned in this track: retrieval metrics to verify the RAG pipeline finds the right chunks, LLM-as-judge scores to verify answer quality, load testing to verify latency SLAs, and security scanning to verify hardening. All must pass before deployment begins.
Running the Full Evaluation Harness
Execute the complete evaluation suite against your staging environment using a representative sample of production-like queries. Record all metrics: hit rate, MRR, NDCG for retrieval; faithfulness and answer relevance for generation; per-query cost; p50/p95/p99 latency. Compare every metric to the success criteria defined in the architecture phase. Do not ship until all hard minimums are met.
async def final_evaluation(system_url: str, test_set_path: str) -> dict:
test_cases = load_test_set(test_set_path)
results = []
for case in test_cases:
start = time.perf_counter()
response = await query_system(system_url, case['question'])
latency_ms = (time.perf_counter() - start) * 1000
judge_score = await judge(case['question'], response['answer'], case.get('reference'))
hit = any(case['relevant_doc'] in s for s in response.get('sources', []))
results.append({'latency_ms': latency_ms, 'score': judge_score, 'hit': hit, 'cost': response.get('cost_usd', 0)})
return compute_final_metrics(results)Load Testing the Production System
Run a load test that simulates realistic production traffic before going live. Use a tool like Locust or k6 to ramp up to your expected peak concurrency (say, 50 simultaneous users) and measure how latency changes under load. Verify that the circuit breakers do not trip under normal load, the rate limiter returns clean 429s under burst traffic, and the semantic cache hit rate remains above your target. Fix any regressions before proceeding.
# Locust load test (locustfile.py)
from locust import HttpUser, task, between
import random
QUESTIONS = [
'What is the return policy?',
'How do I cancel my subscription?',
'Where are you located?',
]
class AIUser(HttpUser):
wait_time = between(1, 3) # realistic think time
@task
def query(self):
self.client.post(
'/query',
json={'question': random.choice(QUESTIONS), 'tenant_id': 'load_test'},
headers={'Authorization': 'Bearer test_token'}
)
# Run: locust -f locustfile.py --headless -u 50 -r 5 --run-time 5mDeploying to Production
Deploy using a blue-green deployment: bring up the new version (green) alongside the existing version (blue), run smoke tests against green, then gradually shift traffic from blue to green. Start with 5% of traffic to green, monitor error rates and latency for 10 minutes, then shift to 25%, then 50%, then 100%. This allows instant rollback to blue if anything goes wrong without any downtime.
# Deployment steps (pseudocode for AWS ECS or K8s):
# 1. Build and push new Docker image
# docker build -t qa-assistant:v2.0 . && docker push ...
#
# 2. Deploy green (new) version alongside blue (current)
# kubectl apply -f deploy/green.yaml
#
# 3. Run smoke tests against green
# pytest tests/smoke/ --base-url https://green.internal
#
# 4. Canary traffic shift (ALB weighted routing)
# 5% -> green (monitor 10min)
# 25% -> green (monitor 10min)
# 50% -> green (monitor 10min)
# 100% -> green
#
# 5. Decommission blue after 24h stabilityPost-Deploy Monitoring
After deploying, watch the key metrics actively for the first 2 hours. Monitor: error rate (target <1%), p95 latency (target <8s), cache hit rate (target >20%), and cost per query (target <$0.05). Set up a war room Slack channel with all engineers on call for the first deploy. Any metric crossing a warning threshold triggers investigation; a critical threshold triggers immediate rollback to blue.
# Post-deploy monitoring dashboard queries:
# (Assuming Grafana + Prometheus)
# Error rate (last 5 min):
# rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m])
# p95 latency (last 5 min):
# histogram_quantile(0.95, rate(query_duration_seconds_bucket[5m]))
# Cache hit rate:
# rate(cache_hits_total[5m]) / rate(queries_total[5m])
# Average cost per query:
# rate(llm_cost_usd_total[5m]) / rate(queries_total[5m])Writing the Architecture Retrospective
After the system has been live for a week, write a retrospective document that captures what worked, what did not, and what you would do differently. A good retrospective is honest about failures and specific about learnings. Future readers — including you six months from now — will benefit from understanding the reasoning behind decisions made under time pressure and the unexpected obstacles encountered.
# RETROSPECTIVE.md structure:
#
# ## What Worked Well
# - Hybrid retrieval improved hit rate from 71% to 89%
# - Semantic cache reduced average cost by 31%
# - LangSmith tracing saved 2 days of debugging
#
# ## What Did Not Work
# - Semantic chunking was 4x slower than recursive chunking
# with only 3% hit rate improvement -- not worth it
# - Cohere reranker had 400ms latency -- too slow for p95 target
# Switched to BGE-reranker-v2 running locally
#
# ## What We'd Do Differently
# - Start with pgvector, not Pinecone (migration cost 3 days)
# - Add semantic cache BEFORE building the agent, not afterDocumenting Operational Runbooks
Write runbooks for every alert that can fire in production. A runbook is a step-by-step guide for diagnosing and resolving a specific alert. It should answer: what does this alert mean, what is the likely cause, how do I diagnose it, and how do I fix it. Runbooks reduce mean time to resolution (MTTR) from hours to minutes by eliminating the need to think through diagnosis steps during a stressful incident.
# runbooks/latency.md
# ## Alert: p95 Latency > 8000ms
#
# ### Likely Causes
# 1. OpenAI API degraded (check status.openai.com)
# 2. Cohere reranker slow (check Cohere status page)
# 3. pgvector query slow (check DB CPU in CloudWatch)
# 4. Redis cache full (check Redis memory usage)
#
# ### Diagnosis
# curl https://api.openai.com/v1/models -H 'Authorization: Bearer $KEY'
# Check LangSmith traces for which step is slow
# SELECT mean(duration) FROM traces GROUP BY step
#
# ### Fixes
# - If OpenAI slow: circuit breaker should auto-failover to Claude
# - If Cohere slow: disable reranking temporarily (env SKIP_RERANK=1)
# - If DB slow: increase pgvector ef_search from 40 to 20Measuring Business Impact
After one month in production, measure the system's business impact beyond technical metrics. For a Q&A assistant: how much did customer support ticket volume decrease? What is the user satisfaction score on AI-answered queries? How many queries did the system handle that previously required human support agents? These business metrics justify the investment and guide future prioritization between quality improvements and new features.
# Business impact metrics (month 1):
# Technical:
# - 12,847 queries served, 11,439 (89%) answered without human
# - Avg query cost: $0.031 (within $0.05 budget)
# - System uptime: 99.94%
#
# Business:
# - Support tickets: 1,840/month -> 1,203/month (-35%)
# - Avg resolution time: 4.2h -> 23 seconds for AI-answered
# - User CSAT on AI answers: 4.1/5.0
# - Cost per resolved query: $12 (human) -> $0.031 (AI)
# - ROI: 157% in month 1 at current volumesPlanning the Next Iteration
A shipped system is never finished — it is a foundation for continuous improvement. Use your evaluation data, user feedback, and retrospective learnings to plan the next iteration. Prioritize improvements that have the highest impact on the metrics that matter most: if retrieval hit rate is the limiting factor, invest in better chunking or a different embedding model; if user satisfaction is low despite good retrieval, invest in prompt quality.
# Next iteration priorities (based on first month data):
NEXT_SPRINT = [
# High impact / high confidence
{
'feature': 'Sentence-window retrieval',
'expected_impact': 'Hit rate 89% -> 93%',
'effort': 'medium',
'evidence': '11% of failures due to answer split across chunks'
},
# High impact / medium confidence
{
'feature': 'Query decomposition for multi-hop questions',
'expected_impact': 'Multi-hop correctness 61% -> 78%',
'effort': 'high',
'evidence': '23% of failures are multi-hop questions'
},
# Low effort quick win
{
'feature': 'Extend cache TTL from 1h to 24h',
'expected_impact': 'Cache hit rate 31% -> 38%',
'effort': 'trivial',
'evidence': 'Same questions asked daily by different users'
}
]Knowledge Sharing and Documentation
Write documentation for the system's key non-obvious implementation decisions. These include: why the specific chunk size was chosen (and what experiments showed), how the semantic cache similarity threshold was calibrated, which injection patterns the filter currently catches and which it does not, and how to add new tools to the agent. Good internal documentation reduces onboarding time for new contributors and prevents decisions from being accidentally reversed by someone who does not know the rationale.
# INTERNALS.md — key non-obvious decisions:
#
# ## Chunk Size: 800 tokens with 100 token overlap
# We tested 400, 600, 800, 1200 tokens.
# 800 tokens maximizes hit rate (89%) while keeping context
# small enough for 5 chunks to fit comfortably in 4096 token prompt.
# Larger chunks improved recall but degraded precision.
#
# ## Cache Similarity Threshold: 0.92
# Tested 0.85, 0.90, 0.92, 0.95.
# 0.92 gives 31% hit rate with <2% incorrect cache hits.
# 0.85 gives 41% hit rate but 8% incorrect hits (too aggressive).
#
# ## Reranker Top-N: 3 (from initial 10)
# More than 3 chunks causes context stuffing without quality gain.What You Have Built
Reflect on the complete system you have built across this capstone: a hybrid RAG pipeline combining dense and sparse retrieval with reranking, a streaming agent with function calling and safety guardrails, a semantic cache reducing costs by 30%, circuit breakers providing automatic failover, LLM-as-judge evaluation running continuously in CI/CD, and prompt injection defenses protecting against adversarial inputs. This is production AI engineering.
# System capabilities summary:
SYSTEM_CAPABILITIES = {
'retrieval': 'Hybrid BM25+dense with Cohere reranking, 89% hit rate',
'generation': 'GPT-4o with Claude fallback, circuit breaker, streaming',
'caching': 'Semantic cache (Redis+embeddings), 31% cache hit rate',
'security': 'Injection filter (2-stage) + output scanning + tenant isolation',
'observability': 'LangSmith traces + Prometheus metrics + PagerDuty alerts',
'evaluation': 'Automated LLM-as-judge in CI, daily full suite, LangSmith evals',
'reliability': '99.94% uptime, circuit breakers, graceful degradation ladder',
'cost': '$0.031/query average, model routing saves 67% vs GPT-4o only',
}Quick Check
Test your understanding of production deployment and evaluation for AI systems.
Lesson Recap
In this lesson you learned: final evaluation combines retrieval metrics, LLM-as-judge scores, load testing, and security scanning before any deployment begins, blue-green deployment with gradual traffic shifting allows instant rollback without downtime, and retrospectives and runbooks capture institutional knowledge that reduces future incident resolution time. Congratulations on completing the AI Engineering: LLM, RAG and Agents track!
常见问题解答
「评估、部署与复盘」课时是免费的吗？
是的 — 「评估、部署与复盘」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。
「评估、部署与复盘」这节课中我会学到什么？
运行完整的评估框架，包括检索指标、LLM-as-judge 质量评分和负载测试；将应用部署到云服务商，并撰写复盘报告，记录经验教训。 你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 AI Engineering Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「评估、部署与复盘」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 AI Engineering Academy 课中编写并运行代码吗？
能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。