Run Multiple Model Instances per GPU
Use concurrent execution to lift utilization.
Run Multiple Model Instances per GPU is a free MLOps Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
One Copy Can Stall
With a single model copy, request two must wait while request one runs. Even a fast GPU can sit idle between calls, leaving throughput on the table.
Run Several Copies
Triton can load multiple instances of the same model so several requests execute concurrently and overlap their work on the GPU.
The instance_group Block
You declare copies with an instance_group in config.pbtxt. The count field says how many instances Triton should create for that model.
instance_group {
count: 2
kind: KIND_GPU
}Pick GPU or CPU
The kind field chooses the device. KIND_GPU runs instances on the GPU, while KIND_CPU runs them on the host processor instead.
Place Them on GPUs
You can pin instances to specific cards with a gpus list. This lets one model spread copies across several GPUs in the same server.
instance_group {
count: 2
kind: KIND_GPU
gpus: [ 0, 1 ]
}Why It Helps
While one instance does math, another can load inputs or copy results. This overlap hides idle gaps and lifts overall utilization.
It Pairs With Batching
Instances and dynamic batching work together. Batching fills each call, while multiple instances keep more than one call in flight at once.
Watch the Memory
Each instance holds its own copy of the weights in GPU memory. Too many copies and you run out of VRAM, so raise the count gradually.
More Is Not Always Faster
Past a point, extra instances just compete for the same compute. Throughput plateaus or drops, so the best count comes from measuring, not guessing.
Concurrency in Mind
The right instance count depends on how many requests arrive at once. Match instances to your real concurrency to avoid both stalls and waste.
A Sensible Starting Point
Two instances per GPU is a common starting point. Test with realistic load, then adjust the count up or down based on what you observe.
Quick Check
What does setting count to 2 in an instance_group do?
Recap
You learned to run several model copies via instance_group, pairing instances with batching for parallelism, while watching VRAM and tuning the count by measurement. 🙌
Frequently asked questions
Is the “Run Multiple Model Instances per GPU” lesson free?
Yes — the full text of “Run Multiple Model Instances per GPU” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Run Multiple Model Instances per GPU”?
Use concurrent execution to lift utilization. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Run Multiple Model Instances per GPU” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why GPUs Need Batching
- Configure Dynamic Batching in Triton
- Run Multiple Model Instances per GPU
- Profile and Tune Inference Latency