Keep a Human in the Loop
Add approval steps for high-stakes models.
Keep a Human in the Loop is a free MLOps Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Automation Has Limits
Full auto-promotion is great until a model affects money, safety, or people. For high stakes, keep a human in the loop before anything ships.
Human-in-the-Loop
Human-in-the-loop means a person reviews and approves a key step. The pipeline pauses and waits for a deliberate yes.
Why Pause for Approval
Metrics miss context: fairness, legal risk, or a weird edge case. A reviewer adds judgment that a threshold check simply cannot.
The Approval Gate
You insert a manual approval step after evaluation and before promotion. Nothing reaches production until someone signs off.
Pause a DAG to Wait
In Airflow you can halt a run until a human acts. This sensor waits for an external approval signal before continuing.
wait = ExternalTaskSensor(
task_id="await_approval",
external_task_id="human_signed_off")Give Reviewers Evidence
Show the reviewer the metrics, drift report, and sample predictions. Good context turns approval from a guess into a real decision.
Make Rejection Easy
A reviewer must be able to say no and roll the pipeline back cleanly. Rejection should be a first-class, low-friction outcome.
Set an Approval Timeout
Do not let a run hang forever waiting on a person. A timeout fails the run safely so nothing silently stalls in limbo.
Notify the Right People
Send a Slack or email ping when approval is needed. Fast notifications stop the pipeline from waiting on someone who never saw it.
Tier by Risk
Reserve manual review for high-risk models; let low-stakes ones auto-promote. Matching oversight to risk keeps you both safe and fast.
Record Who Approved
Log the approver, the time, and their note for every promotion. This accountability trail is essential for audits and compliance.
Quick Check
Where does a human-in-the-loop approval step usually sit?
Recap
You added oversight: an approval gate after evaluation, rich context for reviewers, timeouts, notifications, and a logged sign-off. Automate the safe, review the risky. ✅
Frequently asked questions
Is the “Keep a Human in the Loop” lesson free?
Yes — the full text of “Keep a Human in the Loop” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Keep a Human in the Loop”?
Add approval steps for high-stakes models. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Keep a Human in the Loop” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.