Profile and Tune Inference Latency

You cannot tune what you do not measure. Before changing settings, capture real latency and throughput numbers under load you trust. 📏

Triton ships with perf_analyzer, a tool that hammers your model with requests and reports latency and throughput so you can compare configs.

You point perf_analyzer at a model and it sends synthetic requests, then prints inferences per second and the latency breakdown.

The most useful flag sweeps concurrency, the number of in-flight requests. It reveals how throughput and latency change as load rises.

As concurrency climbs, throughput rises then flattens while latency keeps growing. The knee of that curve is your practical operating point.

Averages hide pain. Watch the p95 latency, the value 95 percent of requests beat, because tail latency is what users actually feel.

If latency is too high, lower max_queue_delay_microseconds. If throughput is too low under heavy load, raise it to build fuller batches.

If the GPU is underused, add an instance. If memory is tight or throughput stalls, drop one. Change one knob at a time and re-measure.

Tuning is a loop, not a leap. Adjust a single setting, rerun perf_analyzer, compare, and keep the change only if the numbers actually improve.

Decide your budget first, like p95 under 50 ms. Then push batch size and instances as far as you can while staying inside that limit.

Server numbers are not the full story. Network hops and client code add time, so also measure end-to-end latency from where the user sits.

Why prefer p95 latency over average latency when tuning?

You used perf_analyzer to sweep concurrency, read the throughput-latency curve at p95, and tuned queue delay and instances within a budget, one change at a time. 🙌