0Pricing
Machine Learning Academy · Lesson

Choosing the Right Metric for Your Business Problem

Learners will analyse three real scenarios (fraud detection, medical diagnosis, house pricing) and justify the best evaluation metric for each.

Choosing the Right Metric for Your Business Problem is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Metrics Are Business Decisions

The choice of evaluation metric is not a purely technical decision — it reflects your business priorities and cost structure. A fraud detection model optimised for accuracy will underperform on detecting fraud in an imbalanced dataset. A medical screening model optimised for precision might miss too many sick patients. Every metric implicitly assumes something about what types of errors are acceptable. Defining the right metric before building the model is one of the most important decisions in any ML project. The metric drives all subsequent choices: model selection, threshold tuning, and deployment criteria.

# The metric choice depends on error costs

# Scenario 1: Email spam filter
# FP (ham marked as spam) -> user misses important email: HIGH cost
# FN (spam reaches inbox) -> user is annoyed: LOW cost
# -> Optimise PRECISION (avoid false alarms)

# Scenario 2: Cancer screening
# FP (healthy patient flagged) -> unnecessary follow-up: LOW cost
# FN (cancer patient missed) -> delayed treatment: HIGH cost
# -> Optimise RECALL (catch all true cases)

# Scenario 3: House price prediction
# No asymmetric cost -> use RMSE or R^2
print('Define your error cost structure BEFORE choosing a metric')

Scenario 1: Fraud Detection

Fraud detection is the classic imbalanced binary classification problem. Legitimate transactions outnumber fraud by 1000:1 or more. A model predicting 'legitimate' for every transaction achieves 99.9% accuracy but is completely useless. The right primary metric depends on business constraints: if fraud losses vastly exceed investigation costs, optimise recall (catch all fraud, even at the cost of many false positives). If customer experience is critical, balance recall and precision. The Precision-Recall curve and AUC-PR are preferred over ROC for such extreme imbalance.

import numpy as np
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    average_precision_score, roc_auc_score
)

# Simulated fraud: 1000 transactions, 10 are fraud (1%)
y_true = np.array([0]*990 + [1]*10)

# Model A: always predict legitimate
y_dummy = np.zeros(1000, dtype=int)

print('Dummy model (always legitimate):')
print(f'  Accuracy: {accuracy_score(y_true, y_dummy):.3f}')  # 99% -- misleading!
print(f'  Recall:   {recall_score(y_true, y_dummy):.3f}')    # 0.0 -- catches no fraud!
print()
print('For fraud detection: use Recall, F1, or AUC-PR')

Scenario 2: Medical Diagnosis

In medical diagnosis, missing a true disease case (false negative) is far more costly than an unnecessary follow-up test (false positive). Recall (sensitivity) should be the primary metric: you want to catch as many true disease cases as possible. A high-recall, lower-precision model is acceptable — patients with false positives undergo additional testing and are cleared; patients with false negatives miss their diagnosis entirely. The threshold is typically set very low (e.g., 0.1 probability) to maximise recall, at the cost of many false alarms that are filtered by follow-up tests.

from sklearn.metrics import recall_score, precision_score, f1_score
import numpy as np

# Simulated diagnosis: 1000 patients, 50 have the disease
y_true = np.array([0]*950 + [1]*50)

# High-recall model: flags 200 positives, 45 are true positive
y_high_recall = np.array([0]*900 + [1]*100)  # approximate
y_pred_labels = np.zeros(1000, dtype=int)
y_pred_labels[-100:] = 1  # flag last 100 (including 50 true)

TP = 50  # catches all 50 true positives
FP = 50  # but also flags 50 healthy patients
FN = 0   # misses no true positives

recall = TP / (TP + FN)
prec   = TP / (TP + FP)
print(f'Medical screening model: Recall={recall:.2f}, Precision={prec:.2f}')
print('Recall 1.0 is ideal: no cancer patient missed.')

Scenario 3: E-commerce Recommendation

Recommendation systems present users with a ranked list of items. The goal is to maximise the quality of recommendations. Key metrics: Precision@K — of the top K recommendations, what fraction are relevant? Recall@K — of all relevant items, what fraction appear in the top K? NDCG (Normalised Discounted Cumulative Gain) weights relevance by position — a relevant item in position 1 is worth more than one in position 10. Click-through rate and purchase conversion are business KPIs that ultimately matter more than any ML metric — always tie model metrics to business outcomes.

import numpy as np

def precision_at_k(recommended, relevant, k):
    '''Of top k recommendations, fraction that are relevant'''
    top_k = recommended[:k]
    relevant_set = set(relevant)
    hits = sum(1 for item in top_k if item in relevant_set)
    return hits / k

# Example: recommend 10 items, 3 are truly relevant
recommended = ['item_A', 'item_B', 'item_C', 'item_D', 'item_E',
               'item_F', 'item_G', 'item_H', 'item_I', 'item_J']
relevant     = ['item_A', 'item_C', 'item_G']

for k in [3, 5, 10]:
    print(f'Precision@{k}: {precision_at_k(recommended, relevant, k):.2f}')

Scenario 4: House Price Prediction

For regression tasks like predicting house prices, the metric choice depends on the use case. A real estate platform showing estimated prices to buyers cares about MAE — the typical dollar error. A lender using predictions for risk assessment cares more about not underestimating prices significantly, suggesting a directional metric. For model comparison, R^2 provides a normalised measure. If predictions feed into another system (e.g., investment algorithm), the downstream impact on returns is the true metric — and you must simulate the business process to measure it.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

# Simulated house prices (in $1000s)
y_true = np.array([250, 380, 510, 420, 650, 290, 730, 480])
y_pred = np.array([265, 360, 525, 435, 620, 310, 700, 465])

mae  = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
r2   = r2_score(y_true, y_pred)

print('House Price Model Performance:')
print(f'  MAE:  ${mae:.1f}k -- typical error for a buyer seeing estimates')
print(f'  RMSE: ${rmse:.1f}k -- influenced by occasional large errors')
print(f'  R^2:  {r2:.3f} -- fraction of price variance explained')

Business KPIs vs ML Metrics: Bridging the Gap

ML metrics measure statistical properties of predictions, but business KPIs measure actual business outcomes. The goal is to maximise business value, using ML metrics as a proxy. Common disconnects: a model might improve AUC from 0.85 to 0.88 but not actually increase revenue if the improvement is in low-value predictions. Conversely, a small recall improvement for high-value fraud cases could save millions. Always connect ML metric improvements to estimated business value: 'raising recall by 5% catches 50 additional fraud cases per month worth $X'.

# Translating ML metric improvements to business value

# Fraud detection example:
new_cases_per_month = 10000   # Fraud transactions per month
avg_fraud_value = 500         # Average fraud value ($)
current_recall = 0.70         # Model catches 70% of fraud
proposed_recall = 0.80        # Improved model catches 80%

current_caught  = new_cases_per_month * current_recall  * avg_fraud_value
proposed_caught = new_cases_per_month * proposed_recall * avg_fraud_value

business_value = proposed_caught - current_caught
print(f'Recall improvement: {current_recall:.0%} -> {proposed_recall:.0%}')
print(f'Business value: ${business_value:,.0f}/month additional fraud prevented')

Metric Alignment: Training Loss vs Evaluation Metric

A subtle but important issue: the metric you train on (the loss function) may differ from the metric you evaluate on. Logistic regression minimises cross-entropy loss, not F1-score. If your evaluation metric is F1, the model may not be optimally aligned. Solutions: use threshold optimisation after training to find the threshold that maximises F1 on validation data; or use algorithms that directly optimise the target metric (e.g., XGBoost with custom objectives). For regression, LinearRegression minimises MSE — if you care about MAE, consider using HuberRegressor or MAE-specific loss functions.

from sklearn.metrics import f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

# Find threshold that maximises F1
best_f1, best_thresh = 0, 0.5
for thresh in np.linspace(0.1, 0.9, 81):
    pred = (proba >= thresh).astype(int)
    f1 = f1_score(y_te, pred, zero_division=0)
    if f1 > best_f1:
        best_f1, best_thresh = f1, thresh

print(f'Default (0.5) F1:  {f1_score(y_te, (proba>=0.5).astype(int)):.4f}')
print(f'Optimised F1:      {best_f1:.4f} at threshold {best_thresh:.2f}')

Multi-Metric Reporting: The Complete Picture

In production ML projects, never report a single metric — provide a metric dashboard that gives a complete picture. For classification: accuracy (to see overall), precision and recall (to understand error types), F1 (as balanced summary), and AUC (threshold-independent). For regression: MAE, RMSE, and R^2 together. Include baseline comparisons: a DummyClassifier or DummyRegressor that uses the simplest strategy (always predict majority class or mean). Every real model must substantially outperform the baseline on every relevant metric to be considered useful.

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
)
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
X_tr_s = StandardScaler().fit_transform(X_tr)
X_te_s = StandardScaler().fit(X_tr).transform(X_te)

for name, clf in [('Dummy', DummyClassifier(strategy='most_frequent')),
                   ('LogReg', LogisticRegression(max_iter=1000))]:
    clf.fit(X_tr_s, y_tr)
    pred = clf.predict(X_te_s)
    prob = clf.predict_proba(X_te_s)[:, 1]
    print(f'\n{name}:')
    print(f'  Accuracy={accuracy_score(y_te,pred):.3f}, '
          f'Precision={precision_score(y_te,pred):.3f}, '
          f'Recall={recall_score(y_te,pred):.3f}, '
          f'F1={f1_score(y_te,pred):.3f}, '
          f'AUC={roc_auc_score(y_te,prob):.3f}')

Calibration: Do Probabilities Mean What They Say?

A model might have good AUC but poor calibration — its predicted probabilities do not match true frequencies. A well-calibrated model saying 80% probability should be correct 80% of the time. Calibration matters when: the probability itself is used (not just the class label), in risk-scoring systems, in multi-stage pipelines where one model's output is another's input. Use CalibrationDisplay to plot predicted probability bins against actual positive fraction. Random forests and boosted trees are often poorly calibrated; logistic regression and Naive Bayes tend to be better. Post-hoc calibration with CalibratedClassifierCV can fix this.

from sklearn.calibration import CalibrationDisplay
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

fig, ax = plt.subplots(figsize=(8, 6))
for name, clf in [('LogReg', LogisticRegression(max_iter=1000)),
                   ('RandomForest', RandomForestClassifier(n_estimators=100))]:
    clf.fit(X_tr, y_tr)
    CalibrationDisplay.from_estimator(clf, X_te, y_te, ax=ax, name=name, n_bins=10)

ax.set_title('Calibration Curves')
plt.show()

A Decision Framework for Metric Selection

Here is a complete decision framework: Step 1: Is this classification or regression? Step 2: For classification — are classes balanced? If not, use AUC/F1/PR curve. Step 3: What is the relative cost of FP vs FN? High FP cost → precision; high FN cost → recall. Step 4: For regression — are errors scale-dependent? If so, use RMSE/MAE; if comparing across problems, use R^2. Step 5: Do probabilities need to be accurate? Add calibration evaluation. Step 6: Define a business KPI translation for the metric. Document this framework in every project's model card.

def select_metric(task, balanced, fp_cost_high, fn_cost_high, need_proba_cal):
    if task == 'regression':
        return 'MAE + RMSE + R^2 (report all three)'
    # Classification:
    if not balanced:
        if fn_cost_high and fp_cost_high:
            return 'F1 + AUC-PR (balance precision and recall)'
        elif fn_cost_high:
            return 'Recall + AUC-PR (minimise missed positives)'
        elif fp_cost_high:
            return 'Precision + AUC-ROC (minimise false alarms)'
    else:
        return 'Accuracy + F1 + AUC-ROC'
    return 'AUC-ROC + F1'

print(select_metric('classification', False, False, True, False))  # Recall
print(select_metric('classification', True, True, True, False))    # Accuracy+F1
print(select_metric('regression', True, False, False, False))      # MAE+RMSE+R^2

Documenting the Metric Choice

Every deployed model should have a model card that documents the chosen metric, the rationale, the baseline score, the deployed model's score, and the business KPI translation. This creates accountability: when business stakeholders ask 'is the model good enough?', you can answer with 'our fraud model achieves 82% recall on transactions over $1000 at a 3% false positive rate, preventing $2.3M in monthly losses compared to our rule-based system.' This is the difference between an academic experiment and a production ML system. Metric documentation is as important as the model itself.

# Minimal model card template
model_card = {
    'model_name': 'Fraud Detector v2.1',
    'task': 'Binary classification (fraud / legitimate)',
    'primary_metric': 'Recall at FPR=0.03',
    'metric_rationale': 'Missing fraud (FN) costs $500 avg; false alarm (FP) costs $5 investigation',
    'baseline': {'model': 'DummyClassifier(most_frequent)', 'recall': 0.0, 'fpr': 0.0},
    'deployed_model': {'algorithm': 'XGBoost', 'recall': 0.82, 'fpr': 0.03},
    'business_kpi': '$2.3M/month fraud prevented vs rule-based system',
    'evaluation_date': '2026-06-20',
    'training_data': 'Transactions 2024-01 to 2025-12, 5M samples'
}

for key, value in model_card.items():
    print(f'{key}: {value}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: metric selection is a business decision driven by the relative costs of false positives and false negatives, how three real scenarios — fraud detection, medical diagnosis, and recommendation systems — require different primary metrics, and the importance of documenting metric choice and connecting it to business KPIs. This completes the Model Evaluation Metrics course — next up we begin the MIDLEVEL track with Random Forest and Ensemble Methods.

Frequently asked questions

Is the “Choosing the Right Metric for Your Business Problem” lesson free?

Yes — the full text of “Choosing the Right Metric for Your Business Problem” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Choosing the Right Metric for Your Business Problem”?

Learners will analyse three real scenarios (fraud detection, medical diagnosis, house pricing) and justify the best evaluation metric for each. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Choosing the Right Metric for Your Business Problem” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Classification Metrics: Accuracy, Precision, Recall, F1
  2. ROC Curves and AUC-ROC
  3. Regression Metrics: MAE, MSE, RMSE, and R-Squared
  4. Choosing the Right Metric for Your Business Problem
← Back to Machine Learning Academy