Run Training as a Kubernetes Job
Execute batch training to completion on the cluster.
Run Training as a Kubernetes Job is a free MLOps Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Training Is Not a Server
A model server runs forever, but a training run should start, finish, and stop. Kubernetes has a different object built for that: the Job. 🏁
A Job Runs to Completion
A Job creates one or more Pods and watches them until they exit successfully. Once training succeeds, the Job is done and frees its resources.
apiVersion: batch/v1
kind: Job
metadata:
name: train-ranker
spec:
template:
spec:
restartPolicy: NeverrestartPolicy Must Not Be Always
A Job Pod must use restartPolicy Never or OnFailure. Always is for servers and Kubernetes rejects it for a Job that is meant to end.
Retries with backoffLimit
Training can fail on a flaky download. The backoffLimit sets how many times the Job retries a failing Pod before it gives up for good.
spec:
backoffLimit: 4
activeDeadlineSeconds: 3600Cap Runtime to Avoid Runaways
Set activeDeadlineSeconds so a hung training run cannot burn an expensive GPU node all weekend. The Job is killed once that limit passes. ⏱️
Request the GPU You Need
A training Job uses the same resources block as a Deployment. Add nvidia.com/gpu under limits so the run lands on a GPU node.
Parallelism for Sweeps
Set completions and parallelism to run many Pods, perfect for a hyperparameter sweep where each Pod trains one configuration at once.
Logs and Artifacts Outlive the Pod
A finished Job Pod is gone, so write your model and metrics to durable storage like S3 or a volume, never to the Pod filesystem.
CronJob for Scheduled Retraining
Wrap a Job in a CronJob to retrain on a schedule, like nightly. It creates a fresh Job each time the cron expression fires.
apiVersion: batch/v1
kind: CronJob
spec:
schedule: "0 2 * * *"Clean Up Finished Jobs
Completed Jobs linger by default and clutter the cluster. Set ttlSecondsAfterFinished so Kubernetes deletes them automatically after a grace period.
Watch Status with kubectl
Check a run with kubectl get jobs and read output with kubectl logs. The status shows succeeded or failed counts at a glance.
kubectl get jobs
kubectl logs job/train-rankerQuick Check
Why is a Job, not a Deployment, right for training?
Recap
Run finite training as a Job with restartPolicy Never, a backoffLimit, and a runtime cap, then schedule retraining with a CronJob. ✅
Frequently asked questions
Is the “Run Training as a Kubernetes Job” lesson free?
Yes — the full text of “Run Training as a Kubernetes Job” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Run Training as a Kubernetes Job”?
Execute batch training to completion on the cluster. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Run Training as a Kubernetes Job” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Pods, Deployments, and Services for Models
- Request CPU, Memory, and GPU
- Configure with ConfigMaps and Secrets
- Run Training as a Kubernetes Job