负载测试与容量规划
学习如何对 LLM 应用模拟真实流量，找出其承载极限，并规划容量，使生产环境在负载下保持快速运行且不超出预算。
负载测试与容量规划 是 CoddyKit 上的免费 LLM Apps in Production (RAG + Vector DB + Caching) 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 LLM Apps in Production (RAG + Vector DB + Caching) 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 LLM Apps in Production (RAG + Vector DB + Caching) 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Why Load Test LLM Apps?
LLM apps behave differently under load than typical web services: token generation is slow, requests are long-lived, and upstream provider rate limits add a hard ceiling.
Load testing reveals how your system degrades before real users do.
Key Metrics
Track these under load:
- Throughput — requests or tokens per second
- Latency percentiles — p50, p95, p99
- Error rate — timeouts, 429s
- Time to first token for streaming
Open vs Closed Load Models
Two ways to generate load:
- Closed — fixed number of virtual users, each waits for a response before sending the next
- Open — requests arrive at a fixed rate regardless of responses
Open-model tests better expose queue buildup.
Realistic Workloads
Use realistic prompts. A test with tiny prompts hides cost; production prompts include long retrieved context. Sample real queries and vary input length to mimic actual token distributions.
Percentile Latency in Code
Averages lie; percentiles tell the truth about tail latency.
def percentile(values, p):
s = sorted(values)
idx = int(round((p/100) * (len(s)-1)))
return s[idx]
lat = [120, 130, 140, 900, 150]
print('p95 =', percentile(lat, 95))Finding the Breaking Point
Ramp the request rate gradually until latency or error rate crosses your SLO. That inflection point is your saturation capacity. Run below it in production with headroom.
Estimating Required Capacity
Use Little's Law: concurrency = arrival rate x average latency. Estimate how many concurrent slots you need for peak traffic.
def concurrency(rps, avg_latency_s):
return rps * avg_latency_s
print('Need', concurrency(50, 2.0), 'concurrent slots')Accounting for Provider Limits
Your effective capacity may be capped by the LLM provider's tokens-per-minute and requests-per-minute limits, not your servers. Plan around those quotas and request increases ahead of launches.
Headroom and Autoscaling
Run at a target utilization (often 60-70 percent) so spikes do not immediately saturate. Configure autoscaling on a leading signal like queue depth, since CPU is a poor proxy for LLM load.
Soak and Spike Tests
Beyond steady ramps, run:
- Soak — sustained load for hours to catch leaks
- Spike — sudden surge to test autoscaling reaction
From Test to Plan
Turn results into a capacity plan: peak rps, required concurrency, provider quota needs, scaling rules, and a cost estimate. Re-test after major changes since model and prompt changes shift the numbers.
Quick Check
Test your understanding of capacity planning.
Recap
You learned to load test LLM apps with realistic workloads, track latency percentiles and error rate, find the saturation point, and size capacity with Little's Law. Account for provider quotas, keep headroom, autoscale on queue depth, and run soak and spike tests.
用 AI 导师学习 LLM Apps in Production (RAG + Vector DB + Caching) — 免费
在浏览器中编写并运行真实代码，获得全天候 AI 导师的即时帮助，并在网页或应用中继续学习。
- 课程
- 12
- 课程
- 48
常见问题解答
「负载测试与容量规划」课时是免费的吗？
是的 — 「负载测试与容量规划」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 LLM Apps in Production (RAG + Vector DB + Caching) 课程的其余内容，请升级到 CoddyKit PRO。 LLM Apps in Production (RAG + Vector DB + Caching) 课程共包含 4 节课。
「负载测试与容量规划」这节课中我会学到什么？
学习如何对 LLM 应用模拟真实流量，找出其承载极限，并规划容量，使生产环境在负载下保持快速运行且不超出预算。 你通过在浏览器中直接运行的动手代码来练习 LLM Apps in Production (RAG + Vector DB + Caching)，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 LLM Apps in Production (RAG + Vector DB + Caching) 需要有经验吗？
无需任何先前经验。CoddyKit 上的 LLM Apps in Production (RAG + Vector DB + Caching) 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「负载测试与容量规划」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 LLM Apps in Production (RAG + Vector DB + Caching) 课中编写并运行代码吗？
能。每节 LLM Apps in Production (RAG + Vector DB + Caching) 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。