Classification Metrics: Accuracy, Precision, Recall, F1

Accuracy — the fraction of correct predictions — is the most intuitive metric, but it is misleading when classes are imbalanced. Consider a medical test for a disease that affects 1% of patients. A model that always predicts 'healthy' achieves 99% accuracy while being completely useless — it misses every actual disease case. In fraud detection, cancer screening, or rare event prediction, accuracy hides the model's failure to detect the rare but critical positive class. This is why we need metrics that separately measure performance on each class: precision, recall, and F1-score.

The confusion matrix is the foundation of all classification metrics. For binary classification, it is a 2x2 table: True Positives (TP) — correctly predicted positive; True Negatives (TN) — correctly predicted negative; False Positives (FP) — predicted positive but actually negative (Type I error); False Negatives (FN) — predicted negative but actually positive (Type II error). Every classification metric is derived from some combination of these four numbers. Inspecting the raw confusion matrix reveals which type of error the model makes most.

Accuracy = (TP + TN) / (TP + TN + FP + FN). It answers: of all predictions, what fraction were correct? Accuracy is a fair metric only when classes are roughly balanced and the costs of false positives and false negatives are similar. For example, handwritten digit recognition with ~10% of each digit class is balanced, making accuracy a reasonable measure. Use accuracy when: classes are balanced and all errors have equal cost. Avoid it when: classes are skewed (one class dominates) or false negatives cost more than false positives (or vice versa).

Precision = TP / (TP + FP). It answers: of all the samples we predicted as positive, what fraction actually are positive? Precision is the metric to optimise when false positives are costly. In spam detection: a high-precision spam filter rarely marks legitimate email as spam. In drug testing: high precision means we rarely approve a drug that does not work. Low precision means many of our positive predictions are wrong — we are flooding users with false alarms. Precision tells you nothing about the false negatives — a model that only predicts one obvious case as positive has precision 1.0 but misses everything else.

Recall = TP / (TP + FN) (also called Sensitivity or True Positive Rate). It answers: of all samples that actually are positive, what fraction did we find? Recall is the metric to optimise when false negatives are costly. In cancer screening: high recall means we identify most patients who actually have cancer. In fraud detection: high recall means we catch most fraudulent transactions. Low recall means we are missing too many actual positives — dangerous in medical or security contexts. Recall and precision are in tension: increasing one typically decreases the other.

Precision and recall are inversely related when you adjust the classification threshold. As you lower the threshold (predict positive more aggressively), recall increases (you catch more actual positives) but precision decreases (more false positives get through). Raising the threshold has the opposite effect. There is no single threshold that simultaneously maximises both. Which metric to prioritise depends on the business problem: in email filtering, prioritise precision (do not annoy users with false spam alerts); in disease screening, prioritise recall (do not miss sick patients).

F1-score = 2 * (Precision * Recall) / (Precision + Recall). The F1-score is the harmonic mean of precision and recall. Unlike the arithmetic mean, the harmonic mean punishes extreme values: a model with precision=1.0 and recall=0.0 gets F1=0, not 0.5. This makes F1 a fair aggregate metric that requires both precision and recall to be reasonably high. F1 is the standard metric for imbalanced text classification (NLP benchmarks always report it), fraud detection, and any problem where class imbalance makes accuracy misleading.

The Fbeta score generalises F1 by introducing a parameter beta that controls how much to weight recall relative to precision: F_beta = (1 + beta^2) * Precision * Recall / (beta^2 * Precision + Recall). When beta=1, this is the standard F1 score. When beta=2 (F2), recall is weighted twice as much as precision — useful when missing positives is twice as costly as false alarms. When beta=0.5 (F0.5), precision is weighted more — useful in recommendation systems where accuracy of suggestions matters more than coverage.

For multi-class classification, you must aggregate per-class precision/recall/F1 into a single number. Macro average: compute metric for each class independently, then average — treats all classes equally regardless of size. Weighted average: same but weights by class support — gives larger classes more influence. Micro average: aggregate TP/FP/FN across all classes first, then compute — equivalent to accuracy for precision, recall, and F1 when there are no samples with multiple labels. Use macro for imbalanced datasets where small minority classes are important.

classification_report() generates a formatted table showing precision, recall, F1-score, and support (number of true instances) for each class, plus macro/weighted averages. This is the standard output to include in any ML evaluation report. The 'support' column is especially important: a class with support=5 will have noisy precision/recall estimates and should be interpreted cautiously. Always check support values alongside F1 scores to assess metric reliability.

Balanced accuracy is the arithmetic mean of sensitivity (recall for each class). For binary classification: balanced_accuracy = (TPR + TNR) / 2 where TNR (True Negative Rate) = TN / (TN+FP). Unlike standard accuracy, balanced accuracy gives equal weight to each class regardless of how many samples it has. A model that always predicts the majority class gets balanced accuracy of 0.5 — correctly identifying it as no better than random. Use balanced accuracy when classes are imbalanced and you want a single metric that reflects performance on all classes equally, without the complexity of computing per-class metrics.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: accuracy is misleading for imbalanced classes — use the confusion matrix as ground truth, precision measures quality of positive predictions (minimise false positives) and recall measures coverage of actual positives (minimise false negatives), and F1-score harmonically combines both into a balanced metric. Next up we explore ROC curves and AUC-ROC for threshold-independent model evaluation.