0Pricing
Learn AI with Python · Lesson

Scaling and Auto-Scaling Model Endpoints

Kubernetes HPA for model pods, traffic-based scaling, A/B deployment, canary releases.

Scaling and Auto-Scaling Model Endpoints is a free Learn AI with Python lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Auto-Scaling

Inference traffic is rarely flat. Provisioning for peak load wastes money at night; provisioning for average load fails at peak. Auto-scaling adjusts the number of replicas automatically based on demand so you pay for what you use and stay responsive.

Pods and Replicas

On Kubernetes a model server runs as a Deployment of identical pods (replicas). Scaling means changing the replica count. A Service load-balances requests across whatever replicas exist.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: server
          image: my-model:latest

Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) watches a metric and adds or removes pods to keep it near a target. The classic metric is CPU utilization.

HPA on CPU Utilization

targetCPUUtilizationPercentage tells the HPA to keep average CPU near a target. If pods average 80% CPU and the target is 50%, the HPA scales up until utilization falls back toward 50%.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-server
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

Limits of CPU-Based Scaling

CPU is a poor proxy for some workloads. A GPU model may bottleneck on the GPU while CPU stays low, or an async worker may have a growing queue while CPU is idle. For these, scale on a workload-specific signal instead.

KEDA for Event-Driven Scaling

KEDA (Kubernetes Event-Driven Autoscaling) scales on external metrics such as queue length, Kafka lag, or Prometheus queries. It can even scale to zero when there is no work, which HPA cannot do on its own.

KEDA Queue-Based ScaledObject

This KEDA ScaledObject adds workers when a message queue grows. Each unit of queueLength backlog triggers another replica, so the consumer keeps up with bursts.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference-worker
spec:
  scaleTargetRef:
    name: inference-worker
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
    - type: rabbitmq
      metadata:
        queueName: inference
        queueLength: "10"

Load Balancing with Ingress

External traffic reaches your pods through an ingress. An nginx ingress controller terminates TLS and distributes requests across the Service backing your replicas, so scaling up immediately spreads load.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: model-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /predict
            pathType: Prefix
            backend:
              service:
                name: model-server
                port:
                  number: 8000

Safe Rollouts: Canary

Deploying a new model version to all traffic at once is risky. A canary release sends a small slice of traffic to the new version, watches its metrics, then gradually shifts more over.

Canary with Traffic Weights

The nginx ingress canary annotations route a percentage of requests to a second deployment. Start at 10%, verify error rate and latency, then raise the weight to 100%.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"

Putting Scaling Together

A robust setup combines all of these:

  • HPA or KEDA sets the replica count by demand
  • Ingress load-balances across replicas
  • Canary weights roll out new versions safely

Together they give elastic, resilient, low-risk model serving.

Quick Check

Test your scaling knowledge.

Recap

You learned to scale model endpoints:

  • HPA with targetCPUUtilizationPercentage scales on CPU
  • KEDA scales on queue length and supports scale-to-zero
  • nginx ingress load-balances across replicas
  • Canary weights shift traffic gradually for safe releases

Frequently asked questions

Is the “Scaling and Auto-Scaling Model Endpoints” lesson free?

Yes — the full text of “Scaling and Auto-Scaling Model Endpoints” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Scaling and Auto-Scaling Model Endpoints”?

Kubernetes HPA for model pods, traffic-based scaling, A/B deployment, canary releases. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Scaling and Auto-Scaling Model Endpoints” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Containerizing ML Models with Docker
  2. Cloud Deployment: AWS SageMaker
  3. High-Performance Serving with Triton Inference Server
  4. Scaling and Auto-Scaling Model Endpoints
← Back to Learn AI with Python