レイテンシー、スループット、コストのトレードオフ

Every serving choice juggles three things: latency, throughput, and cost. Push hard on one and you usually move the other two.

Latency is the time for a single prediction to come back. Low latency feels snappy; high latency makes users and downstream systems wait.

Throughput is how many predictions you serve per second. A service can be fast per call yet still need high throughput under heavy load.

Grouping requests into a batch raises throughput but adds wait time, so each call sees higher latency. The two goals often fight.

More machines cut latency and lift throughput, but the bill climbs. Cost is the third corner you cannot ignore when sizing a service.

An SLA sets your target, like 95% of requests under 100 ms. It turns vague goals into a number you design and measure against.

Serving many inputs in one model call uses hardware better. This batching lifts throughput, ideal when a little extra latency is fine.

Add more replicas to share traffic. Horizontal scaling raises throughput and protects latency, at the price of more compute spend.

Averages hide pain. The slow p99 request is what users complain about, so you tune for the tail, not just the typical case.

A GPU can crush throughput on big models but sits idle on light traffic. Match the hardware to your real load to avoid wasted cost.

There is no universal best. You weigh latency, throughput, and cost against what your users truly need, then choose deliberately.

You enable request batching. What usually happens?

Latency, throughput, and cost form a triangle you cannot max all at once. Set an SLA, then use batching and scaling to hit the balance you need.