Exécuter plusieurs instances de modèle par GPU

With a single model copy, request two must wait while request one runs. Even a fast GPU can sit idle between calls, leaving throughput on the table.

Triton can load multiple instances of the same model so several requests execute concurrently and overlap their work on the GPU.

You declare copies with an instance_group in config.pbtxt. The count field says how many instances Triton should create for that model.

The kind field chooses the device. KIND_GPU runs instances on the GPU, while KIND_CPU runs them on the host processor instead.

You can pin instances to specific cards with a gpus list. This lets one model spread copies across several GPUs in the same server.

While one instance does math, another can load inputs or copy results. This overlap hides idle gaps and lifts overall utilization.

Instances and dynamic batching work together. Batching fills each call, while multiple instances keep more than one call in flight at once.

Each instance holds its own copy of the weights in GPU memory. Too many copies and you run out of VRAM, so raise the count gradually.

Past a point, extra instances just compete for the same compute. Throughput plateaus or drops, so the best count comes from measuring, not guessing.

The right instance count depends on how many requests arrive at once. Match instances to your real concurrency to avoid both stalls and waste.

Two instances per GPU is a common starting point. Test with realistic load, then adjust the count up or down based on what you observe.

What does setting count to 2 in an instance_group do?

You learned to run several model copies via instance_group, pairing instances with batching for parallelism, while watching VRAM and tuning the count by measurement. 🙌