Dlaczego GPU potrzebują batchingu

A GPU has thousands of cores built to do the same math on many items at once. Feed it one input and almost all of those cores sit idle. 🐢

Serving a single prediction per call barely touches the hardware. The GPU spends more time waiting than computing, so your expensive card is mostly idle.

A batch stacks many inputs into one tensor and runs them in a single pass. The GPU does roughly the same work, but for many requests at once.

Two numbers matter here. Throughput is how many predictions per second you serve; latency is how long one request waits for its answer.

Grouping requests raises throughput dramatically because fixed per-call overhead is shared. You get far more predictions from the same GPU.

There is a catch. To form a batch the server must wait briefly for more requests to arrive, which adds a little latency to each one.

The simplest form is static batching, where the client itself sends a fixed-size batch. It works for offline jobs but not for live, one-at-a-time traffic.

Dynamic batching lets the server group separate single requests on the fly. Triton Inference Server can do this for you with no client changes.

NVIDIA Triton Inference Server hosts models and includes a scheduler that forms batches automatically to keep the GPU fully fed.

A well-batched GPU serves many more users per dollar. Batching is often the cheapest way to cut your inference bill before buying more hardware. 💰

Your job is to keep the GPU busy without making any single user wait too long. The rest of this course tunes that balance in Triton.

Why does running one input at a time waste a GPU?

You saw that GPUs need many inputs at once. Batching groups requests to lift throughput for a small latency cost, and Triton can batch dynamically for you. 🙌