Enable Adaptive Micro-Batching
Group requests automatically for higher throughput.
Enable Adaptive Micro-Batching is a free MLOps Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Throughput Problem
Calling a model once per request wastes hardware. Models run far faster on a batch of inputs than on the same inputs one at a time. ⚡
What Micro-Batching Does
Adaptive batching collects incoming requests for a brief window, runs them together, then splits results back to each caller automatically.
Why Adaptive
The window size is not fixed. BentoML watches live latency and adapts the batch size so it stays fast under both light and heavy load.
It Lives on the Runner
Batching is configured per model, not per request. You enable it on the runnable by marking which methods support batched inputs.
Turn It On
For a custom runnable you set batchable to true on the method. BentoML then groups calls to that method behind the scenes.
@bentoml.Runnable.method(batchable=True)
def predict(self, inputs):
...Pick the Batch Axis
BentoML needs to know how to stack inputs. The batch_dim argument tells it which axis to concatenate along, usually axis 0.
@bentoml.Runnable.method(batchable=True, batch_dim=0)Cap the Batch Size
You bound how big a batch can grow. max_batch_size caps the number of requests merged so one giant batch never stalls others.
Cap the Wait Time
You also bound how long to wait. max_latency_ms sets the longest a request may sit in the queue before the batch fires.
Tune It in Config
You set these limits without touching code. A bentoml_configuration file lets you adjust batching per runner for each environment.
runners:
predict:
batching:
max_batch_size: 32The Trade-off
Bigger batches lift throughput but add a little latency per request. Tuning means finding the sweet spot for your traffic.
Watch It Work
You do not change your client at all. Callers still send single requests while BentoML merges them under the hood transparently.
Quick Check
You raise max_batch_size to a large value. What is the likely effect on a single request?
Recap
You learned that adaptive batching groups requests on a batchable runner, tuned by max_batch_size and max_latency_ms, trading a little latency for big throughput. 🙌
Frequently asked questions
Is the “Enable Adaptive Micro-Batching” lesson free?
Yes — the full text of “Enable Adaptive Micro-Batching” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Enable Adaptive Micro-Batching”?
Group requests automatically for higher throughput. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Enable Adaptive Micro-Batching” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Save a Model to the Bento Store
- Define a Service and Its API
- Enable Adaptive Micro-Batching
- Build a Bento and Containerize It