Configure Dynamic Batching in Triton

Triton loads models from a model repository, a folder where each model has its own subfolder holding the weights and a config file.

Each model gets a config.pbtxt next to its version folder. This text file tells Triton the inputs, outputs, and how to schedule requests.

You turn batching on by adding a dynamic_batching block to config.pbtxt. With an empty block, Triton uses sensible defaults right away.

At the top level you set max_batch_size. It caps how many requests Triton will merge into one inference call, bounding memory and latency.

The key knob is max_queue_delay_microseconds. It is how long Triton waits, gathering requests, before it fires a batch that is not yet full.

A short delay keeps latency low but forms smaller batches under light load. A longer delay builds bigger batches at the cost of more waiting.

You can hint efficient shapes with preferred_batch_size. Triton tries to assemble batches of these sizes, which often match what the model runs fastest.

Triton holds incoming requests in a queue. It fires when the batch hits max size, matches a preferred size, or the queue delay expires.

The best part is that clients still send single requests. Triton merges them on the server, so your application code stays exactly the same.

After editing config.pbtxt you reload the model so Triton picks up the new schedule, either by restarting or via the model control API.

Begin with a modest max_batch_size and a tiny queue delay, then raise them while watching latency. Tuning by measurement beats guessing.

Which config field controls how long Triton waits before firing a partial batch?

You enabled dynamic_batching in config.pbtxt, capped it with max_batch_size, and tuned the queue delay and preferred sizes, all without touching the client. 🙌