Right-Size Instances and Replicas
Match hardware to real load profiles.
Right-Size Instances and Replicas is a free MLOps Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Pay for What You Use
Most ML serving bills come from instances sitting half-idle. Right-sizing means matching hardware and replica count to your real load, not your worst-case fear.
Measure Before You Cut
You cannot right-size what you have not measured. Start by watching real CPU and memory utilization over a normal traffic day.
The Overprovisioning Trap
Picking a giant instance just in case feels safe but burns cash every hour. Chronically low utilization is the clearest sign of overprovisioning.
Read Your Utilization
If average CPU sits near 15 percent, your instance is far too big. Aim for a healthy target utilization with headroom for short spikes.
kubectl top pods -n servingVertical vs Horizontal
Vertical scaling gives one replica a bigger machine. Horizontal scaling adds more small replicas instead, which usually handles bursty traffic better.
Set Requests and Limits
On Kubernetes, your resource request tells the scheduler how much each replica truly needs. Set it from observed usage, not a round guess.
resources:
requests:
cpu: "500m"
memory: "1Gi"Pick the Right Replica Count
Too few replicas means queued requests and slow responses. Too many means idle pods you still pay for. Size replicas to your peak concurrent load.
Let Autoscaling Track Demand
A Horizontal Pod Autoscaler adds replicas when load rises and removes them when it falls, so you stop paying for off-peak capacity.
kubectl autoscale deploy model --min 2 --max 10 --cpu-percent 60Keep a Safety Floor
A minimum replica count keeps a few pods warm so traffic never hits a cold, empty service. This is your trade-off between cost and availability.
GPU Boxes Are Pricey
GPU instances cost many times more than CPU. Only request a GPU when your model truly needs it, and pack work tightly so it never sits idle.
Right-Sizing Is Ongoing
Traffic patterns drift over weeks and months. Revisit instance type and replica counts on a schedule so your fleet stays a good fit.
Quick Check
Let us see what low utilization is telling you.
Recap
You measured utilization, chose between vertical and horizontal scaling, set requests, and added autoscaling. Your fleet now matches real demand. 💸
Frequently asked questions
Is the “Right-Size Instances and Replicas” lesson free?
Yes — the full text of “Right-Size Instances and Replicas” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Right-Size Instances and Replicas”?
Match hardware to real load profiles. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Right-Size Instances and Replicas” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Right-Size Instances and Replicas
- Quantize and Distill for Cheaper Inference
- Use Spot Instances for Training
- Track Cost per Prediction