Classification Metrics Deep Dive
Confusion matrix, precision, recall, F1, ROC-AUC, PR curve — choosing the right metric.
Classification Metrics Deep Dive is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Accuracy Is Not Enough
Accuracy is the fraction of correct predictions, but it misleads on imbalanced data. A model predicting only the majority class can score 99% while being useless for the rare class.
The Confusion Matrix
The confusion matrix breaks predictions into true positives, false positives, true negatives, and false negatives. It is the foundation for all other metrics.
from sklearn.metrics import confusion_matrix
y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]
print(confusion_matrix(y_true, y_pred))
# rows = actual, columns = predictedPrecision
Precision = TP / (TP + FP). Of all items predicted positive, how many truly are? High precision means few false alarms. Important when false positives are costly (e.g. spam filters).
from sklearn.metrics import precision_score
print(precision_score(y_true, y_pred))Recall
Recall = TP / (TP + FN). Of all actual positives, how many did we catch? High recall means few misses. Important when false negatives are costly (e.g. disease screening).
from sklearn.metrics import recall_score
print(recall_score(y_true, y_pred))F1 Score
The F1 score is the harmonic mean of precision and recall: 2 * P * R / (P + R). It balances both and is useful when you need a single number on imbalanced data.
from sklearn.metrics import f1_score, classification_report
print(f1_score(y_true, y_pred))
print(classification_report(y_true, y_pred))ROC Curve and AUC
The ROC curve plots true positive rate against false positive rate across thresholds. roc_auc_score summarizes it: 1.0 is perfect, 0.5 is random guessing.
from sklearn.metrics import roc_auc_score
# needs probability scores, not hard labels
proba = model.predict_proba(Xte)[:, 1]
print(roc_auc_score(yte, proba))Precision-Recall Curve
The PR curve plots precision vs recall across thresholds. It is more informative than ROC when classes are heavily imbalanced, because it focuses on the positive class.
from sklearn.metrics import precision_recall_curve
prec, rec, thresh = precision_recall_curve(yte, proba)
# plot rec (x) vs prec (y) to see the tradeoffAverage Precision
average_precision_score summarizes the PR curve into one number (area under the PR curve). It is the preferred summary metric for imbalanced binary classification.
from sklearn.metrics import average_precision_score
print(average_precision_score(yte, proba))ROC AUC vs Average Precision
ROC AUC can look optimistic on imbalanced data because true negatives dominate. Average precision ignores true negatives and reflects performance on the rare positive class more honestly.
Choosing the Right Metric
Use F1 or average precision for imbalanced problems. Use ROC AUC for balanced ranking quality. Pick precision when false alarms hurt, recall when misses hurt. Always state your priority before choosing.
Threshold Tuning
Most metrics depend on the decision threshold (default 0.5). By moving the threshold you trade precision for recall. Use the PR curve to pick a threshold that meets your business requirement.
import numpy as np
# pick threshold for at least 0.9 precision
idx = np.argmax(prec >= 0.9)
chosen = thresh[idx]
preds = (proba >= chosen).astype(int)Quick Check
Test your metrics understanding.
Recap
Recap: Start from the confusion_matrix. Precision penalizes false alarms, recall penalizes misses, F1 balances them. roc_auc_score measures ranking quality; average_precision_score and the PR curve are better for imbalanced data. Tune the threshold to match your priorities.
Frequently asked questions
Is the “Classification Metrics Deep Dive” lesson free?
Yes — the full text of “Classification Metrics Deep Dive” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Classification Metrics Deep Dive”?
Confusion matrix, precision, recall, F1, ROC-AUC, PR curve — choosing the right metric. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Classification Metrics Deep Dive” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Cross-Validation Strategies
- Classification Metrics Deep Dive
- Grid Search and Random Search
- Bayesian Optimization with Optuna