Adaptives Micro-Batching aktivieren

Calling a model once per request wastes hardware. Models run far faster on a batch of inputs than on the same inputs one at a time. ⚡

Adaptive batching collects incoming requests for a brief window, runs them together, then splits results back to each caller automatically.

The window size is not fixed. BentoML watches live latency and adapts the batch size so it stays fast under both light and heavy load.

Batching is configured per model, not per request. You enable it on the runnable by marking which methods support batched inputs.

For a custom runnable you set batchable to true on the method. BentoML then groups calls to that method behind the scenes.

BentoML needs to know how to stack inputs. The batch_dim argument tells it which axis to concatenate along, usually axis 0.

You bound how big a batch can grow. max_batch_size caps the number of requests merged so one giant batch never stalls others.

You also bound how long to wait. max_latency_ms sets the longest a request may sit in the queue before the batch fires.

You set these limits without touching code. A bentoml_configuration file lets you adjust batching per runner for each environment.

Bigger batches lift throughput but add a little latency per request. Tuning means finding the sweet spot for your traffic.

You do not change your client at all. Callers still send single requests while BentoML merges them under the hood transparently.

You raise max_batch_size to a large value. What is the likely effect on a single request?

You learned that adaptive batching groups requests on a batchable runner, tuned by max_batch_size and max_latency_ms, trading a little latency for big throughput. 🙌