중요한 지표 선택하기

A challenger can win on offline accuracy yet lose money. The metric that decides your A/B test should be the business outcome you actually care about. 💡

Offline metrics like F1 or AUC come from a static test set. Online metrics come from live user behavior, and only those tell you the real production impact.

Choose a single primary metric up front, like conversion rate or revenue per user. Deciding after you see the data is how teams fool themselves.

A strong primary metric is tied to value, moves with the model, and is measurable per user. Click-through, purchases, and watch time are classic examples.

Track guardrail metrics too: things that must not get worse, like latency, error rate, or churn. A win that breaks a guardrail is not a real win.

The lift is how much the challenger improves over the champion on your primary metric, usually as a percent change.

A vanity metric looks great but drives no value, like raw page views. Pretty numbers that do not connect to the business can mislead a whole team.

Clicks are an easy proxy for satisfaction, but users can click then bounce. Make sure your proxy truly tracks the outcome you really want.

An average can hide trouble. A model may lift new users but hurt loyal ones, so check the metric across key segments, not just the whole.

Some outcomes, like a refund or a renewal, arrive days later. Pick a metric whose signal lands within your test window, or you will decide too early.

Lock your primary metric, guardrails, and success threshold before the test starts. A written plan stops you from cherry-picking a flattering number later. ✅

Let us see which metric should decide the test.

Choose one primary business metric before the test, guard it with guardrails, and avoid vanity proxies. Segment your results and respect delayed signals. 🎯