Pick Metrics That Matter
Choose business KPIs, not just offline accuracy.
Pick Metrics That Matter is a free MLOps Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Accuracy Is Not the Goal
A challenger can win on offline accuracy yet lose money. The metric that decides your A/B test should be the business outcome you actually care about. 💡
Offline vs Online Metrics
Offline metrics like F1 or AUC come from a static test set. Online metrics come from live user behavior, and only those tell you the real production impact.
Pick One Primary Metric
Choose a single primary metric up front, like conversion rate or revenue per user. Deciding after you see the data is how teams fool themselves.
A Good Primary Metric
A strong primary metric is tied to value, moves with the model, and is measurable per user. Click-through, purchases, and watch time are classic examples.
Guardrail Metrics
Track guardrail metrics too: things that must not get worse, like latency, error rate, or churn. A win that breaks a guardrail is not a real win.
Computing the Lift
The lift is how much the challenger improves over the champion on your primary metric, usually as a percent change.
def lift(challenger_rate, champion_rate):
return (challenger_rate - champion_rate) / champion_rate * 100Beware Vanity Metrics
A vanity metric looks great but drives no value, like raw page views. Pretty numbers that do not connect to the business can mislead a whole team.
Watch for Proxy Gaps
Clicks are an easy proxy for satisfaction, but users can click then bounce. Make sure your proxy truly tracks the outcome you really want.
Segment Before You Trust
An average can hide trouble. A model may lift new users but hurt loyal ones, so check the metric across key segments, not just the whole.
Mind the Delay
Some outcomes, like a refund or a renewal, arrive days later. Pick a metric whose signal lands within your test window, or you will decide too early.
Write It Down First
Lock your primary metric, guardrails, and success threshold before the test starts. A written plan stops you from cherry-picking a flattering number later. ✅
Quick Check
Let us see which metric should decide the test.
Recap
Choose one primary business metric before the test, guard it with guardrails, and avoid vanity proxies. Segment your results and respect delayed signals. 🎯
Frequently asked questions
Is the “Pick Metrics That Matter” lesson free?
Yes — the full text of “Pick Metrics That Matter” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Pick Metrics That Matter”?
Choose business KPIs, not just offline accuracy. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Pick Metrics That Matter” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.