Machine Learning Academy · 강의

비즈니스 문제에 알맞은 지표 선택

사기 탐지, 의료 진단, 주택 가격 책정이라는 세 가지 실제 상황을 분석하고, 각각에 가장 적합한 평가 지표를 근거와 함께 선택합니다.

레슨 4/413개 단계

비즈니스 문제에 알맞은 지표 선택은(는) CoddyKit의 무료 Machine Learning Academy 강의입니다. 이것은 4개 중 4번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Machine Learning Academy 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Machine Learning Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Metrics Are Business Decisions

The choice of evaluation metric is not a purely technical decision — it reflects your business priorities and cost structure. A fraud detection model optimised for accuracy will underperform on detecting fraud in an imbalanced dataset. A medical screening model optimised for precision might miss too many sick patients. Every metric implicitly assumes something about what types of errors are acceptable. Defining the right metric before building the model is one of the most important decisions in any ML project. The metric drives all subsequent choices: model selection, threshold tuning, and deployment criteria.

# The metric choice depends on error costs

# Scenario 1: Email spam filter
# FP (ham marked as spam) -> user misses important email: HIGH cost
# FN (spam reaches inbox) -> user is annoyed: LOW cost
# -> Optimise PRECISION (avoid false alarms)

# Scenario 2: Cancer screening
# FP (healthy patient flagged) -> unnecessary follow-up: LOW cost
# FN (cancer patient missed) -> delayed treatment: HIGH cost
# -> Optimise RECALL (catch all true cases)

# Scenario 3: House price prediction
# No asymmetric cost -> use RMSE or R^2
print('Define your error cost structure BEFORE choosing a metric')

Scenario 1: Fraud Detection

Fraud detection is the classic imbalanced binary classification problem. Legitimate transactions outnumber fraud by 1000:1 or more. A model predicting 'legitimate' for every transaction achieves 99.9% accuracy but is completely useless. The right primary metric depends on business constraints: if fraud losses vastly exceed investigation costs, optimise recall (catch all fraud, even at the cost of many false positives). If customer experience is critical, balance recall and precision. The Precision-Recall curve and AUC-PR are preferred over ROC for such extreme imbalance.

import numpy as np
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    average_precision_score, roc_auc_score
)

# Simulated fraud: 1000 transactions, 10 are fraud (1%)
y_true = np.array([0]*990 + [1]*10)

# Model A: always predict legitimate
y_dummy = np.zeros(1000, dtype=int)

print('Dummy model (always legitimate):')
print(f'  Accuracy: {accuracy_score(y_true, y_dummy):.3f}')  # 99% -- misleading!
print(f'  Recall:   {recall_score(y_true, y_dummy):.3f}')    # 0.0 -- catches no fraud!
print()
print('For fraud detection: use Recall, F1, or AUC-PR')

Scenario 2: Medical Diagnosis

In medical diagnosis, missing a true disease case (false negative) is far more costly than an unnecessary follow-up test (false positive). Recall (sensitivity) should be the primary metric: you want to catch as many true disease cases as possible. A high-recall, lower-precision model is acceptable — patients with false positives undergo additional testing and are cleared; patients with false negatives miss their diagnosis entirely. The threshold is typically set very low (e.g., 0.1 probability) to maximise recall, at the cost of many false alarms that are filtered by follow-up tests.

from sklearn.metrics import recall_score, precision_score, f1_score
import numpy as np

# Simulated diagnosis: 1000 patients, 50 have the disease
y_true = np.array([0]*950 + [1]*50)

# High-recall model: flags 200 positives, 45 are true positive
y_high_recall = np.array([0]*900 + [1]*100)  # approximate
y_pred_labels = np.zeros(1000, dtype=int)
y_pred_labels[-100:] = 1  # flag last 100 (including 50 true)

TP = 50  # catches all 50 true positives
FP = 50  # but also flags 50 healthy patients
FN = 0   # misses no true positives

recall = TP / (TP + FN)
prec   = TP / (TP + FP)
print(f'Medical screening model: Recall={recall:.2f}, Precision={prec:.2f}')
print('Recall 1.0 is ideal: no cancer patient missed.')

Scenario 3: E-commerce Recommendation

Recommendation systems present users with a ranked list of items. The goal is to maximise the quality of recommendations. Key metrics: Precision@K — of the top K recommendations, what fraction are relevant? Recall@K — of all relevant items, what fraction appear in the top K? NDCG (Normalised Discounted Cumulative Gain) weights relevance by position — a relevant item in position 1 is worth more than one in position 10. Click-through rate and purchase conversion are business KPIs that ultimately matter more than any ML metric — always tie model metrics to business outcomes.

import numpy as np

def precision_at_k(recommended, relevant, k):
    '''Of top k recommendations, fraction that are relevant'''
    top_k = recommended[:k]
    relevant_set = set(relevant)
    hits = sum(1 for item in top_k if item in relevant_set)
    return hits / k

# Example: recommend 10 items, 3 are truly relevant
recommended = ['item_A', 'item_B', 'item_C', 'item_D', 'item_E',
               'item_F', 'item_G', 'item_H', 'item_I', 'item_J']
relevant     = ['item_A', 'item_C', 'item_G']

for k in [3, 5, 10]:
    print(f'Precision@{k}: {precision_at_k(recommended, relevant, k):.2f}')

Scenario 4: House Price Prediction

For regression tasks like predicting house prices, the metric choice depends on the use case. A real estate platform showing estimated prices to buyers cares about MAE — the typical dollar error. A lender using predictions for risk assessment cares more about not underestimating prices significantly, suggesting a directional metric. For model comparison, R^2 provides a normalised measure. If predictions feed into another system (e.g., investment algorithm), the downstream impact on returns is the true metric — and you must simulate the business process to measure it.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

# Simulated house prices (in $1000s)
y_true = np.array([250, 380, 510, 420, 650, 290, 730, 480])
y_pred = np.array([265, 360, 525, 435, 620, 310, 700, 465])

mae  = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
r2   = r2_score(y_true, y_pred)

print('House Price Model Performance:')
print(f'  MAE:  ${mae:.1f}k -- typical error for a buyer seeing estimates')
print(f'  RMSE: ${rmse:.1f}k -- influenced by occasional large errors')
print(f'  R^2:  {r2:.3f} -- fraction of price variance explained')

Business KPIs vs ML Metrics: Bridging the Gap

ML metrics measure statistical properties of predictions, but business KPIs measure actual business outcomes. The goal is to maximise business value, using ML metrics as a proxy. Common disconnects: a model might improve AUC from 0.85 to 0.88 but not actually increase revenue if the improvement is in low-value predictions. Conversely, a small recall improvement for high-value fraud cases could save millions. Always connect ML metric improvements to estimated business value: 'raising recall by 5% catches 50 additional fraud cases per month worth $X'.

# Translating ML metric improvements to business value

# Fraud detection example:
new_cases_per_month = 10000   # Fraud transactions per month
avg_fraud_value = 500         # Average fraud value ($)
current_recall = 0.70         # Model catches 70% of fraud
proposed_recall = 0.80        # Improved model catches 80%

current_caught  = new_cases_per_month * current_recall  * avg_fraud_value
proposed_caught = new_cases_per_month * proposed_recall * avg_fraud_value

business_value = proposed_caught - current_caught
print(f'Recall improvement: {current_recall:.0%} -> {proposed_recall:.0%}')
print(f'Business value: ${business_value:,.0f}/month additional fraud prevented')

Metric Alignment: Training Loss vs Evaluation Metric

A subtle but important issue: the metric you train on (the loss function) may differ from the metric you evaluate on. Logistic regression minimises cross-entropy loss, not F1-score. If your evaluation metric is F1, the model may not be optimally aligned. Solutions: use threshold optimisation after training to find the threshold that maximises F1 on validation data; or use algorithms that directly optimise the target metric (e.g., XGBoost with custom objectives). For regression, LinearRegression minimises MSE — if you care about MAE, consider using HuberRegressor or MAE-specific loss functions.

from sklearn.metrics import f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

# Find threshold that maximises F1
best_f1, best_thresh = 0, 0.5
for thresh in np.linspace(0.1, 0.9, 81):
    pred = (proba >= thresh).astype(int)
    f1 = f1_score(y_te, pred, zero_division=0)
    if f1 > best_f1:
        best_f1, best_thresh = f1, thresh

print(f'Default (0.5) F1:  {f1_score(y_te, (proba>=0.5).astype(int)):.4f}')
print(f'Optimised F1:      {best_f1:.4f} at threshold {best_thresh:.2f}')

Multi-Metric Reporting: The Complete Picture

In production ML projects, never report a single metric — provide a metric dashboard that gives a complete picture. For classification: accuracy (to see overall), precision and recall (to understand error types), F1 (as balanced summary), and AUC (threshold-independent). For regression: MAE, RMSE, and R^2 together. Include baseline comparisons: a DummyClassifier or DummyRegressor that uses the simplest strategy (always predict majority class or mean). Every real model must substantially outperform the baseline on every relevant metric to be considered useful.

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
)
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
X_tr_s = StandardScaler().fit_transform(X_tr)
X_te_s = StandardScaler().fit(X_tr).transform(X_te)

for name, clf in [('Dummy', DummyClassifier(strategy='most_frequent')),
                   ('LogReg', LogisticRegression(max_iter=1000))]:
    clf.fit(X_tr_s, y_tr)
    pred = clf.predict(X_te_s)
    prob = clf.predict_proba(X_te_s)[:, 1]
    print(f'\n{name}:')
    print(f'  Accuracy={accuracy_score(y_te,pred):.3f}, '
          f'Precision={precision_score(y_te,pred):.3f}, '
          f'Recall={recall_score(y_te,pred):.3f}, '
          f'F1={f1_score(y_te,pred):.3f}, '
          f'AUC={roc_auc_score(y_te,prob):.3f}')

Calibration: Do Probabilities Mean What They Say?

A model might have good AUC but poor calibration — its predicted probabilities do not match true frequencies. A well-calibrated model saying 80% probability should be correct 80% of the time. Calibration matters when: the probability itself is used (not just the class label), in risk-scoring systems, in multi-stage pipelines where one model's output is another's input. Use CalibrationDisplay to plot predicted probability bins against actual positive fraction. Random forests and boosted trees are often poorly calibrated; logistic regression and Naive Bayes tend to be better. Post-hoc calibration with CalibratedClassifierCV can fix this.

from sklearn.calibration import CalibrationDisplay
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

fig, ax = plt.subplots(figsize=(8, 6))
for name, clf in [('LogReg', LogisticRegression(max_iter=1000)),
                   ('RandomForest', RandomForestClassifier(n_estimators=100))]:
    clf.fit(X_tr, y_tr)
    CalibrationDisplay.from_estimator(clf, X_te, y_te, ax=ax, name=name, n_bins=10)

ax.set_title('Calibration Curves')
plt.show()

A Decision Framework for Metric Selection

Here is a complete decision framework: Step 1: Is this classification or regression? Step 2: For classification — are classes balanced? If not, use AUC/F1/PR curve. Step 3: What is the relative cost of FP vs FN? High FP cost → precision; high FN cost → recall. Step 4: For regression — are errors scale-dependent? If so, use RMSE/MAE; if comparing across problems, use R^2. Step 5: Do probabilities need to be accurate? Add calibration evaluation. Step 6: Define a business KPI translation for the metric. Document this framework in every project's model card.

def select_metric(task, balanced, fp_cost_high, fn_cost_high, need_proba_cal):
    if task == 'regression':
        return 'MAE + RMSE + R^2 (report all three)'
    # Classification:
    if not balanced:
        if fn_cost_high and fp_cost_high:
            return 'F1 + AUC-PR (balance precision and recall)'
        elif fn_cost_high:
            return 'Recall + AUC-PR (minimise missed positives)'
        elif fp_cost_high:
            return 'Precision + AUC-ROC (minimise false alarms)'
    else:
        return 'Accuracy + F1 + AUC-ROC'
    return 'AUC-ROC + F1'

print(select_metric('classification', False, False, True, False))  # Recall
print(select_metric('classification', True, True, True, False))    # Accuracy+F1
print(select_metric('regression', True, False, False, False))      # MAE+RMSE+R^2

Documenting the Metric Choice

Every deployed model should have a model card that documents the chosen metric, the rationale, the baseline score, the deployed model's score, and the business KPI translation. This creates accountability: when business stakeholders ask 'is the model good enough?', you can answer with 'our fraud model achieves 82% recall on transactions over $1000 at a 3% false positive rate, preventing $2.3M in monthly losses compared to our rule-based system.' This is the difference between an academic experiment and a production ML system. Metric documentation is as important as the model itself.

# Minimal model card template
model_card = {
    'model_name': 'Fraud Detector v2.1',
    'task': 'Binary classification (fraud / legitimate)',
    'primary_metric': 'Recall at FPR=0.03',
    'metric_rationale': 'Missing fraud (FN) costs $500 avg; false alarm (FP) costs $5 investigation',
    'baseline': {'model': 'DummyClassifier(most_frequent)', 'recall': 0.0, 'fpr': 0.0},
    'deployed_model': {'algorithm': 'XGBoost', 'recall': 0.82, 'fpr': 0.03},
    'business_kpi': '$2.3M/month fraud prevented vs rule-based system',
    'evaluation_date': '2026-06-20',
    'training_data': 'Transactions 2024-01 to 2025-12, 5M samples'
}

for key, value in model_card.items():
    print(f'{key}: {value}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

인터랙티브 레슨에서 답해보세요

Lesson Recap

In this lesson you learned: metric selection is a business decision driven by the relative costs of false positives and false negatives, how three real scenarios — fraud detection, medical diagnosis, and recommendation systems — require different primary metrics, and the importance of documenting metric choice and connecting it to business KPIs. This completes the Model Evaluation Metrics course — next up we begin the MIDLEVEL track with Random Forest and Ensemble Methods.

무료로 시작

AI 튜터와 함께 Python을(를) 배우세요 — 무료

브라우저에서 실제 코드를 작성하고 실행하며, 24/7 AI 튜터로부터 즉각적인 도움을 받고, 웹이나 앱에서 중단한 부분부터 계속 학습하세요.

코스: 30
레슨: 120

무료로 학습 시작 전체 코스 보기

자주 묻는 질문

“비즈니스 문제에 알맞은 지표 선택” 강의는 무료인가요?

네 — “비즈니스 문제에 알맞은 지표 선택” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Machine Learning Academy 강의 전체를 잠금 해제할 수 있습니다. Machine Learning Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

“비즈니스 문제에 알맞은 지표 선택”에서 뭘 배우나요?

사기 탐지, 의료 진단, 주택 가격 책정이라는 세 가지 실제 상황을 분석하고, 각각에 가장 적합한 평가 지표를 근거와 함께 선택합니다. 브라우저에서 직접 실행하는 실습 코드로 Machine Learning Academy을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

Machine Learning Academy을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 Machine Learning Academy은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 4번째 강의입니다.

“비즈니스 문제에 알맞은 지표 선택” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 Machine Learning Academy 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 Machine Learning Academy 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← Machine Learning Academy(으)로 돌아가기