Use Spot Instances for Training
Run interruptible jobs at a fraction of the cost.
Use Spot Instances for Training is a free MLOps Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Cheap Compute, One Catch
Cloud providers rent out spare capacity at a deep discount. These spot instances can cost up to ninety percent less than on-demand machines.
They Can Vanish Anytime
The catch is the provider can reclaim a spot instance with little warning. This interruption is why spot is cheap, and why you must plan for it.
Training Tolerates Interruptions
Training is a great fit for spot because it runs in the background, not in front of users. A paused job hurts far less than a dropped live request.
Checkpoint Early and Often
The key habit is saving progress before an interruption strikes. Checkpointing writes model weights and optimizer state to durable storage as you go.
torch.save({"epoch": epoch, "model": model.state_dict()}, "ckpt.pt")Resume Where You Stopped
When a new spot instance starts, load the last checkpoint and keep going. Resuming turns a reclaimed machine into a brief pause, not lost work.
ckpt = torch.load("ckpt.pt")
model.load_state_dict(ckpt["model"])Store Checkpoints Off the Box
Save checkpoints to durable storage like S3, never the local disk. When the instance dies, your progress must survive outside it.
Heed the Termination Notice
Most clouds send a short warning before reclaiming a machine. Catch that termination notice and flush a final checkpoint while you still can.
Spread Across Instance Types
If one machine type runs out, your job stalls. Requesting several types and zones lifts your odds of grabbing cheap capacity.
Keep Serving on Stable Compute
Spot fits training, but not always low-latency serving users depend on. Keep inference on on-demand or reserved capacity for reliability.
Let Tools Manage the Churn
Managed services like SageMaker Managed Spot or Kubernetes can auto-resume jobs for you, hiding most of the interruption pain.
Weigh Savings Against Delay
Spot trades cost for occasional delays. For a deadline-critical run, the safer on-demand price can be worth paying.
Quick Check
Let us make sure spot training stays safe.
Recap
You learned why spot is cheap, checkpointed often to durable storage, resumed after interruptions, and kept serving on stable compute. 💰
Frequently asked questions
Is the “Use Spot Instances for Training” lesson free?
Yes — the full text of “Use Spot Instances for Training” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Use Spot Instances for Training”?
Run interruptible jobs at a fraction of the cost. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Use Spot Instances for Training” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Right-Size Instances and Replicas
- Quantize and Distill for Cheaper Inference
- Use Spot Instances for Training
- Track Cost per Prediction