衡量 LLM 延迟：TTFT 与 TPOT

Measuring LLM latency as a single number is misleading. There are actually two distinct phases: the time until the first token arrives (perceived responsiveness), and the time it takes to generate subsequent tokens (output speed). A model can have great TTFT but slow TPOT, making long responses feel sluggish even though the initial response felt instant.

Time to First Token (TTFT) is the duration from sending the API request to receiving the very first token of the response. It includes network latency, queuing time at the inference server, and prefill time (processing the input tokens). TTFT dominates perceived responsiveness — users notice when nothing appears for more than 1-2 seconds, regardless of how fast tokens stream afterward.

Time Per Output Token (TPOT) is the average time between successive tokens once generation has started. It is calculated as total generation time divided by total output tokens. TPOT determines reading speed — humans read at roughly 250 words per minute, so a TPOT above 100ms per token (10 tokens/second) will feel noticeably slow for long responses.

The relationship between these metrics is: total_latency = TTFT + (output_tokens × TPOT). For a response of 500 tokens at 50ms TPOT, generation takes 25 seconds. For streaming applications, optimize TTFT first — users tolerate slow streaming better than a blank screen. For batch processing without streaming, total latency is what matters, so optimize TPOT by choosing models with faster inference.

TTFT is dominated by prefill cost, which scales with the number of input tokens. A 10,000-token system prompt will have 10x higher TTFT than a 1,000-token prompt, all else being equal. Other factors include server load (queue time), network round-trip time to the API endpoint, and whether prompt caching reduces the effective prefill work. Minimize system prompt length to reduce TTFT.

TPOT is primarily determined by the model size and hardware. Smaller models (GPT-4o-mini) decode much faster than large models (GPT-4o). On self-hosted models, batch size and GPU memory bandwidth are the main factors. For OpenAI-hosted models, TPOT varies with server load but is typically 15-40ms per token. You cannot directly control TPOT on hosted APIs — model selection is your main lever.

Not all endpoints need the same latency target. A chat endpoint has a tight TTFT SLA (users expect <500ms), while a batch summarization endpoint can tolerate multi-second latency. Define explicit SLA targets per endpoint and instrument each one separately. Common targets: interactive chat = p95 TTFT <600ms, document extraction = p95 total <10s, batch = no real-time SLA.

Log TTFT and TPOT for every production request with structured logging. Include the model name, endpoint, prompt token count, output token count, and whether a cache hit occurred. This gives you the data to compute percentile distributions (p50, p95, p99), spot regressions after model updates, and correlate latency spikes with queue depth or provider incidents.

Raw averages are misleading for latency — a few slow outliers inflate the mean without affecting most users. Always report p50, p95, and p99 percentiles. P95 is the most common SLA metric: it means 95% of requests completed within that time. Use NumPy or the statistics module to compute percentiles from your logged latency samples.

Setting an appropriate max_tokens limit reduces worst-case total latency by preventing runaway long responses. If your use case needs at most 200-token answers, set max_tokens=250. This also caps cost. Combine with streaming to ensure users see output immediately while the full response is still generating. Never leave max_tokens unlimited in production endpoints.

When latency is too high, address bottlenecks in priority order. First, enable streaming so users see output immediately even if total latency is high. Second, shorten the system prompt to reduce TTFT. Third, add prompt prefix caching to amortize prefill cost across repeated requests. Fourth, switch to a smaller model if quality allows. Finally, consider self-hosted inference for maximum control over both TTFT and TPOT.

Test your understanding of TTFT and TPOT as LLM latency metrics.

In this lesson you learned: TTFT (Time to First Token) measures perceived responsiveness and scales with input token count, TPOT (Time Per Output Token) determines generation speed and is mainly controlled by model size, and per-endpoint SLA targets with percentile metrics are the right way to monitor latency in production. Next up we implement load balancing across multiple API keys.