Evaluation, Deployment, and Retrospective
Run the full evaluation harness including retrieval metrics, LLM-as-judge quality scores, and load tests, deploy to a cloud provider, and write a retrospective documenting lessons learned.
Evaluation, Deployment, and Retrospective is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Final Mile: Evaluation Before Shipping
A system is not ready to ship until it has been evaluated end-to-end against real-world conditions. The final evaluation phase combines all evaluation techniques learned in this track: retrieval metrics to verify the RAG pipeline finds the right chunks, LLM-as-judge scores to verify answer quality, load testing to verify latency SLAs, and security scanning to verify hardening. All must pass before deployment begins.
Running the Full Evaluation Harness
Execute the complete evaluation suite against your staging environment using a representative sample of production-like queries. Record all metrics: hit rate, MRR, NDCG for retrieval; faithfulness and answer relevance for generation; per-query cost; p50/p95/p99 latency. Compare every metric to the success criteria defined in the architecture phase. Do not ship until all hard minimums are met.
async def final_evaluation(system_url: str, test_set_path: str) -> dict:
test_cases = load_test_set(test_set_path)
results = []
for case in test_cases:
start = time.perf_counter()
response = await query_system(system_url, case['question'])
latency_ms = (time.perf_counter() - start) * 1000
judge_score = await judge(case['question'], response['answer'], case.get('reference'))
hit = any(case['relevant_doc'] in s for s in response.get('sources', []))
results.append({'latency_ms': latency_ms, 'score': judge_score, 'hit': hit, 'cost': response.get('cost_usd', 0)})
return compute_final_metrics(results)Load Testing the Production System
Run a load test that simulates realistic production traffic before going live. Use a tool like Locust or k6 to ramp up to your expected peak concurrency (say, 50 simultaneous users) and measure how latency changes under load. Verify that the circuit breakers do not trip under normal load, the rate limiter returns clean 429s under burst traffic, and the semantic cache hit rate remains above your target. Fix any regressions before proceeding.
# Locust load test (locustfile.py)
from locust import HttpUser, task, between
import random
QUESTIONS = [
'What is the return policy?',
'How do I cancel my subscription?',
'Where are you located?',
]
class AIUser(HttpUser):
wait_time = between(1, 3) # realistic think time
@task
def query(self):
self.client.post(
'/query',
json={'question': random.choice(QUESTIONS), 'tenant_id': 'load_test'},
headers={'Authorization': 'Bearer test_token'}
)
# Run: locust -f locustfile.py --headless -u 50 -r 5 --run-time 5mDeploying to Production
Deploy using a blue-green deployment: bring up the new version (green) alongside the existing version (blue), run smoke tests against green, then gradually shift traffic from blue to green. Start with 5% of traffic to green, monitor error rates and latency for 10 minutes, then shift to 25%, then 50%, then 100%. This allows instant rollback to blue if anything goes wrong without any downtime.
# Deployment steps (pseudocode for AWS ECS or K8s):
# 1. Build and push new Docker image
# docker build -t qa-assistant:v2.0 . && docker push ...
#
# 2. Deploy green (new) version alongside blue (current)
# kubectl apply -f deploy/green.yaml
#
# 3. Run smoke tests against green
# pytest tests/smoke/ --base-url https://green.internal
#
# 4. Canary traffic shift (ALB weighted routing)
# 5% -> green (monitor 10min)
# 25% -> green (monitor 10min)
# 50% -> green (monitor 10min)
# 100% -> green
#
# 5. Decommission blue after 24h stabilityPost-Deploy Monitoring
After deploying, watch the key metrics actively for the first 2 hours. Monitor: error rate (target <1%), p95 latency (target <8s), cache hit rate (target >20%), and cost per query (target <$0.05). Set up a war room Slack channel with all engineers on call for the first deploy. Any metric crossing a warning threshold triggers investigation; a critical threshold triggers immediate rollback to blue.
# Post-deploy monitoring dashboard queries:
# (Assuming Grafana + Prometheus)
# Error rate (last 5 min):
# rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m])
# p95 latency (last 5 min):
# histogram_quantile(0.95, rate(query_duration_seconds_bucket[5m]))
# Cache hit rate:
# rate(cache_hits_total[5m]) / rate(queries_total[5m])
# Average cost per query:
# rate(llm_cost_usd_total[5m]) / rate(queries_total[5m])Writing the Architecture Retrospective
After the system has been live for a week, write a retrospective document that captures what worked, what did not, and what you would do differently. A good retrospective is honest about failures and specific about learnings. Future readers — including you six months from now — will benefit from understanding the reasoning behind decisions made under time pressure and the unexpected obstacles encountered.
# RETROSPECTIVE.md structure:
#
# ## What Worked Well
# - Hybrid retrieval improved hit rate from 71% to 89%
# - Semantic cache reduced average cost by 31%
# - LangSmith tracing saved 2 days of debugging
#
# ## What Did Not Work
# - Semantic chunking was 4x slower than recursive chunking
# with only 3% hit rate improvement -- not worth it
# - Cohere reranker had 400ms latency -- too slow for p95 target
# Switched to BGE-reranker-v2 running locally
#
# ## What We'd Do Differently
# - Start with pgvector, not Pinecone (migration cost 3 days)
# - Add semantic cache BEFORE building the agent, not afterDocumenting Operational Runbooks
Write runbooks for every alert that can fire in production. A runbook is a step-by-step guide for diagnosing and resolving a specific alert. It should answer: what does this alert mean, what is the likely cause, how do I diagnose it, and how do I fix it. Runbooks reduce mean time to resolution (MTTR) from hours to minutes by eliminating the need to think through diagnosis steps during a stressful incident.
# runbooks/latency.md
# ## Alert: p95 Latency > 8000ms
#
# ### Likely Causes
# 1. OpenAI API degraded (check status.openai.com)
# 2. Cohere reranker slow (check Cohere status page)
# 3. pgvector query slow (check DB CPU in CloudWatch)
# 4. Redis cache full (check Redis memory usage)
#
# ### Diagnosis
# curl https://api.openai.com/v1/models -H 'Authorization: Bearer $KEY'
# Check LangSmith traces for which step is slow
# SELECT mean(duration) FROM traces GROUP BY step
#
# ### Fixes
# - If OpenAI slow: circuit breaker should auto-failover to Claude
# - If Cohere slow: disable reranking temporarily (env SKIP_RERANK=1)
# - If DB slow: increase pgvector ef_search from 40 to 20Measuring Business Impact
After one month in production, measure the system's business impact beyond technical metrics. For a Q&A assistant: how much did customer support ticket volume decrease? What is the user satisfaction score on AI-answered queries? How many queries did the system handle that previously required human support agents? These business metrics justify the investment and guide future prioritization between quality improvements and new features.
# Business impact metrics (month 1):
# Technical:
# - 12,847 queries served, 11,439 (89%) answered without human
# - Avg query cost: $0.031 (within $0.05 budget)
# - System uptime: 99.94%
#
# Business:
# - Support tickets: 1,840/month -> 1,203/month (-35%)
# - Avg resolution time: 4.2h -> 23 seconds for AI-answered
# - User CSAT on AI answers: 4.1/5.0
# - Cost per resolved query: $12 (human) -> $0.031 (AI)
# - ROI: 157% in month 1 at current volumesPlanning the Next Iteration
A shipped system is never finished — it is a foundation for continuous improvement. Use your evaluation data, user feedback, and retrospective learnings to plan the next iteration. Prioritize improvements that have the highest impact on the metrics that matter most: if retrieval hit rate is the limiting factor, invest in better chunking or a different embedding model; if user satisfaction is low despite good retrieval, invest in prompt quality.
# Next iteration priorities (based on first month data):
NEXT_SPRINT = [
# High impact / high confidence
{
'feature': 'Sentence-window retrieval',
'expected_impact': 'Hit rate 89% -> 93%',
'effort': 'medium',
'evidence': '11% of failures due to answer split across chunks'
},
# High impact / medium confidence
{
'feature': 'Query decomposition for multi-hop questions',
'expected_impact': 'Multi-hop correctness 61% -> 78%',
'effort': 'high',
'evidence': '23% of failures are multi-hop questions'
},
# Low effort quick win
{
'feature': 'Extend cache TTL from 1h to 24h',
'expected_impact': 'Cache hit rate 31% -> 38%',
'effort': 'trivial',
'evidence': 'Same questions asked daily by different users'
}
]Knowledge Sharing and Documentation
Write documentation for the system's key non-obvious implementation decisions. These include: why the specific chunk size was chosen (and what experiments showed), how the semantic cache similarity threshold was calibrated, which injection patterns the filter currently catches and which it does not, and how to add new tools to the agent. Good internal documentation reduces onboarding time for new contributors and prevents decisions from being accidentally reversed by someone who does not know the rationale.
# INTERNALS.md — key non-obvious decisions:
#
# ## Chunk Size: 800 tokens with 100 token overlap
# We tested 400, 600, 800, 1200 tokens.
# 800 tokens maximizes hit rate (89%) while keeping context
# small enough for 5 chunks to fit comfortably in 4096 token prompt.
# Larger chunks improved recall but degraded precision.
#
# ## Cache Similarity Threshold: 0.92
# Tested 0.85, 0.90, 0.92, 0.95.
# 0.92 gives 31% hit rate with <2% incorrect cache hits.
# 0.85 gives 41% hit rate but 8% incorrect hits (too aggressive).
#
# ## Reranker Top-N: 3 (from initial 10)
# More than 3 chunks causes context stuffing without quality gain.What You Have Built
Reflect on the complete system you have built across this capstone: a hybrid RAG pipeline combining dense and sparse retrieval with reranking, a streaming agent with function calling and safety guardrails, a semantic cache reducing costs by 30%, circuit breakers providing automatic failover, LLM-as-judge evaluation running continuously in CI/CD, and prompt injection defenses protecting against adversarial inputs. This is production AI engineering.
# System capabilities summary:
SYSTEM_CAPABILITIES = {
'retrieval': 'Hybrid BM25+dense with Cohere reranking, 89% hit rate',
'generation': 'GPT-4o with Claude fallback, circuit breaker, streaming',
'caching': 'Semantic cache (Redis+embeddings), 31% cache hit rate',
'security': 'Injection filter (2-stage) + output scanning + tenant isolation',
'observability': 'LangSmith traces + Prometheus metrics + PagerDuty alerts',
'evaluation': 'Automated LLM-as-judge in CI, daily full suite, LangSmith evals',
'reliability': '99.94% uptime, circuit breakers, graceful degradation ladder',
'cost': '$0.031/query average, model routing saves 67% vs GPT-4o only',
}Quick Check
Test your understanding of production deployment and evaluation for AI systems.
Lesson Recap
In this lesson you learned: final evaluation combines retrieval metrics, LLM-as-judge scores, load testing, and security scanning before any deployment begins, blue-green deployment with gradual traffic shifting allows instant rollback without downtime, and retrospectives and runbooks capture institutional knowledge that reduces future incident resolution time. Congratulations on completing the AI Engineering: LLM, RAG and Agents track!
Frequently asked questions
Is the “Evaluation, Deployment, and Retrospective” lesson free?
Yes — the full text of “Evaluation, Deployment, and Retrospective” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Evaluation, Deployment, and Retrospective”?
Run the full evaluation harness including retrieval metrics, LLM-as-judge quality scores, and load tests, deploy to a cloud provider, and write a retrospective documenting lessons learned. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Evaluation, Deployment, and Retrospective” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Designing the Production Architecture
- Implementing Core RAG and Agent Features
- Hardening: Security, Caching, and Reliability
- Evaluation, Deployment, and Retrospective