รันการฝึกเป็นงานของ Kubernetes

A model server runs forever, but a training run should start, finish, and stop. Kubernetes has a different object built for that: the Job. 🏁

A Job creates one or more Pods and watches them until they exit successfully. Once training succeeds, the Job is done and frees its resources.

A Job Pod must use restartPolicy Never or OnFailure. Always is for servers and Kubernetes rejects it for a Job that is meant to end.

Training can fail on a flaky download. The backoffLimit sets how many times the Job retries a failing Pod before it gives up for good.

Set activeDeadlineSeconds so a hung training run cannot burn an expensive GPU node all weekend. The Job is killed once that limit passes. ⏱️

A training Job uses the same resources block as a Deployment. Add nvidia.com/gpu under limits so the run lands on a GPU node.

Set completions and parallelism to run many Pods, perfect for a hyperparameter sweep where each Pod trains one configuration at once.

A finished Job Pod is gone, so write your model and metrics to durable storage like S3 or a volume, never to the Pod filesystem.

Wrap a Job in a CronJob to retrain on a schedule, like nightly. It creates a fresh Job each time the cron expression fires.

Completed Jobs linger by default and clutter the cluster. Set ttlSecondsAfterFinished so Kubernetes deletes them automatically after a grace period.

Check a run with kubectl get jobs and read output with kubectl logs. The status shows succeeded or failed counts at a glance.

Why is a Job, not a Deployment, right for training?

Run finite training as a Job with restartPolicy Never, a backoffLimit, and a runtime cap, then schedule retraining with a CronJob. ✅