ROC Curves and AUC-ROC
Learners will plot the ROC curve by varying the decision threshold, compute AUC, and interpret it as the probability that a random positive outranks a random negative.
ROC Curves and AUC-ROC is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Limitations of Fixed-Threshold Metrics
Precision, recall, and F1-score all depend on a fixed decision threshold (typically 0.5 for binary classifiers). But the optimal threshold varies by application: a spam filter might use 0.9 (strict, to avoid false positives), while a fraud detector might use 0.1 (lenient, to avoid missing fraud). Evaluating a model at only one threshold gives an incomplete picture of its capability. The ROC curve (Receiver Operating Characteristic) solves this by showing performance across all possible thresholds simultaneously, letting you understand the full range of precision-recall trade-offs the model can achieve.
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]
for t in [0.1, 0.3, 0.5, 0.7, 0.9]:
y_pred = (proba >= t).astype(int)
print(f'thresh={t}: prec={precision_score(y_te,y_pred,zero_division=0):.3f}, '
f'rec={recall_score(y_te,y_pred,zero_division=0):.3f}')True Positive Rate and False Positive Rate
The ROC curve is built from two metrics computed at each threshold: True Positive Rate (TPR) (same as recall/sensitivity): TPR = TP / (TP + FN) — what fraction of actual positives did we catch? False Positive Rate (FPR): FPR = FP / (FP + TN) — what fraction of actual negatives did we incorrectly flag? As the threshold decreases, both TPR and FPR increase. The ROC curve plots TPR vs FPR as the threshold varies from 1 (predict nothing positive) to 0 (predict everything positive). A perfect classifier reaches (0, 1) — zero false alarms, all true positives found.
import numpy as np
from sklearn.metrics import confusion_matrix
def tpr_fpr(y_true, y_pred):
TN, FP, FN, TP = confusion_matrix(y_true, y_pred).ravel()
tpr = TP / (TP + FN) # Recall = sensitivity
fpr = FP / (FP + TN) # False positive rate = 1 - specificity
return tpr, fpr
# Example predictions at different thresholds
y_true = np.array([1,1,1,0,0,0,0,0,0,0])
for thresh in [0.3, 0.5, 0.7]:
# Simulated probabilities
proba = np.array([0.9, 0.8, 0.4, 0.6, 0.5, 0.3, 0.2, 0.1, 0.05, 0.01])
y_pred = (proba >= thresh).astype(int)
tpr, fpr = tpr_fpr(y_true, y_pred)
print(f'thresh={thresh}: TPR={tpr:.2f}, FPR={fpr:.2f}')Plotting the ROC Curve
Scikit-learn's roc_curve() computes the (FPR, TPR) pairs across all thresholds in a single call. RocCurveDisplay creates a publication-ready plot. The ROC curve for a random classifier is the diagonal line from (0,0) to (1,1) — it achieves no better than chance at any threshold. A good classifier bows toward the upper-left corner, achieving high TPR with low FPR. The further the curve is from the diagonal, the better the classifier. The area under this curve (AUC) summarises performance in a single number.
from sklearn.metrics import roc_curve, RocCurveDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
RocCurveDisplay.from_estimator(clf, X_te, y_te)
plt.plot([0,1],[0,1],'k--', label='Random classifier')
plt.title('ROC Curve')
plt.legend()
plt.show()AUC-ROC: Area Under the Curve
AUC-ROC (Area Under the ROC Curve) summarises the ROC curve into a single number between 0 and 1. AUC = 0.5: no better than random (diagonal line). AUC = 1.0: perfect classifier (no false positives at any recall level). AUC = 0.0: perfect inverse classifier — every prediction is wrong (useful if you flip predictions). An AUC of 0.8 means that 80% of the time, a randomly selected positive sample receives a higher predicted probability than a randomly selected negative sample. AUC is threshold-independent — it evaluates the ranking quality of the probability scores, not a specific prediction decision.
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]
auc = roc_auc_score(y_te, proba)
print(f'AUC-ROC: {auc:.4f}')
# Interpretation: AUC = P(score(pos) > score(neg))
import numpy as np
pos_scores = proba[y_te == 1]
neg_scores = proba[y_te == 0]
mc_auc = np.mean([p > n for p in pos_scores for n in neg_scores])
print(f'Monte Carlo AUC: {mc_auc:.4f}') # Should match roc_auc_scoreInterpreting AUC Values in Practice
Guidelines for interpreting AUC: 0.5 = random classifier, useless; 0.6-0.7 = poor, but better than chance; 0.7-0.8 = fair — acceptable for some applications; 0.8-0.9 = good — solid production model; 0.9-1.0 = excellent — likely near state-of-the-art for the problem. Beware of AUC inflation: temporal data leakage (using future information), label leakage, or highly imbalanced classes can artificially inflate AUC. Always sanity-check with calibration curves and by inspecting predictions on specific examples.
# AUC interpretation guide
auc_guide = [
(0.50, 0.60, 'Random to poor'),
(0.60, 0.70, 'Poor but predictive'),
(0.70, 0.80, 'Fair'),
(0.80, 0.90, 'Good'),
(0.90, 1.00, 'Excellent'),
]
for low, high, label in auc_guide:
print(f'AUC [{low:.2f}, {high:.2f}): {label}')
# Note: domain matters -- AUC 0.75 might be excellent
# for predicting disease 10 years in advance but poor
# for predicting email spam from clear signalsComparing Multiple Models on the ROC Curve
Overlaying multiple ROC curves on the same plot enables direct model comparison. Each model's curve and AUC value appear together. A model is better if its ROC curve is uniformly above another's at every FPR. If curves cross, neither model dominates — one is better at low false positive rates, the other at high false positive rates, and the business context determines which is preferable. AUC comparison is a quick summary, but always check curves at the operating point you actually intend to use.
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
models = [
('Logistic Reg', LogisticRegression(max_iter=1000)),
('Decision Tree', DecisionTreeClassifier(max_depth=5)),
('KNN', KNeighborsClassifier(n_neighbors=5)),
]
for name, model in models:
model.fit(StandardScaler().fit_transform(X_tr) if name!='Decision Tree' else X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]
fpr, tpr, _ = roc_curve(y_te, proba)
auc = roc_auc_score(y_te, proba)
plt.plot(fpr, tpr, label=f'{name} (AUC={auc:.3f})')
plt.plot([0,1],[0,1],'k--')
plt.legend()
plt.show()The Precision-Recall Curve: Better for Imbalanced Data
For highly imbalanced datasets, the ROC curve can be misleadingly optimistic. When there are very few positives (e.g., 1% fraud rate), even a model with many false positives achieves a low FPR because the denominator (TN+FP) is huge. The Precision-Recall curve plots precision vs recall at different thresholds and is not affected by class imbalance. The area under the PR curve (Average Precision, AP) is a better summary metric for imbalanced problems. Use ROC for balanced classes; use Precision-Recall curve for imbalanced classes.
from sklearn.metrics import PrecisionRecallDisplay, average_precision_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]
ap = average_precision_score(y_te, proba)
print(f'Average Precision (AP): {ap:.4f}')
PrecisionRecallDisplay.from_estimator(clf, X_te, y_te)
plt.title(f'Precision-Recall Curve (AP={ap:.3f})')
plt.show()AUC in Cross-Validation
To get a reliable AUC estimate that is not biased by a single train-test split, use cross_val_score with scoring='roc_auc'. This computes AUC on each fold's validation set and returns all fold scores. The mean is the best estimate; the standard deviation shows stability. High mean AUC with low std indicates a robust model. High AUC with high std suggests the model is sensitive to which samples appear in each fold — consider whether the data has natural groupings (e.g., multiple samples per patient) that should be kept together in folds.
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([
('sc', StandardScaler()),
('clf', LogisticRegression(max_iter=1000))
])
aucs = cross_val_score(pipe, X, y, cv=10, scoring='roc_auc')
print(f'AUC per fold: {aucs.round(3)}')
print(f'Mean AUC: {aucs.mean():.4f}')
print(f'Std AUC: {aucs.std():.4f}')Setting an Operating Point on the ROC Curve
After evaluating a model with the ROC curve, you must choose a specific operating point (a threshold) for production use. Three strategies: (1) Youden's J statistic: choose the threshold that maximises (TPR - FPR) — the point farthest above the diagonal. (2) Business constraint: set maximum allowable FPR (e.g., flag at most 5% of legitimate transactions as fraud) and find the corresponding threshold. (3) Cost optimisation: if you know the cost of FP and FN, choose the threshold that minimises total cost. Document the chosen threshold along with the model for reproducible deployment.
from sklearn.metrics import roc_curve
import numpy as np
fpr, tpr, thresholds = roc_curve(y_te, proba)
# Strategy 1: Youden's J (max TPR - FPR)
youden_j = tpr - fpr
best_idx = np.argmax(youden_j)
best_threshold = thresholds[best_idx]
print(f'Youden threshold: {best_threshold:.3f}')
print(f'TPR={tpr[best_idx]:.3f}, FPR={fpr[best_idx]:.3f}')
# Strategy 2: Max TPR with FPR <= 0.05
allowed_fpr = 0.05
mask = fpr <= allowed_fpr
idx = np.where(mask)[0][-1] # Highest TPR satisfying FPR constraint
print(f'\nAt FPR<={allowed_fpr}: TPR={tpr[idx]:.3f}, threshold={thresholds[idx]:.3f}')Multi-Class ROC-AUC
For multi-class classification, ROC-AUC is extended by computing AUC for each class in a One-vs-Rest (OvR) setup: treat class k as positive and all others as negative, compute AUC, then average across classes. Scikit-learn's roc_auc_score supports this with multi_class='ovr' and average='macro' or average='weighted'. This requires probability estimates from predict_proba(), so the classifier must support probability output. Macro AUC treats all classes equally; weighted AUC accounts for class frequency.
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_tr, y_tr)
proba_all = clf.predict_proba(X_te) # Shape: (n_samples, n_classes)
auc_macro = roc_auc_score(y_te, proba_all, multi_class='ovr', average='macro')
auc_weighted = roc_auc_score(y_te, proba_all, multi_class='ovr', average='weighted')
print(f'Macro AUC (OvR): {auc_macro:.4f}')
print(f'Weighted AUC (OvR): {auc_weighted:.4f}')AUC for Imbalanced Datasets: A Warning
AUC-ROC can be misleadingly optimistic for severely imbalanced datasets. When only 0.1% of transactions are fraud, even a model that captures 80% of fraud cases while flagging 5% of legitimate ones achieves an AUC near 0.9 — but in absolute terms, flagging 5% of millions of legitimate transactions creates enormous operational cost. In such cases, AUC-PR (area under the Precision-Recall curve) is more informative because it focuses only on the positive class performance and is not inflated by the large number of true negatives. For extreme imbalance (<1% positive rate), always report AUC-PR alongside AUC-ROC.
from sklearn.metrics import roc_auc_score, average_precision_score
from sklearn.linear_model import LogisticRegression
import numpy as np
np.random.seed(42)
# Extremely imbalanced: 1% positive rate
n_neg, n_pos = 9900, 100
X = np.vstack([np.random.randn(n_neg, 5),
np.random.randn(n_pos, 5) + 1.5])
y = np.array([0]*n_neg + [1]*n_pos)
clf = LogisticRegression(class_weight='balanced', max_iter=1000)
clf.fit(X, y)
proba = clf.predict_proba(X)[:, 1]
auc_roc = roc_auc_score(y, proba)
auc_pr = average_precision_score(y, proba)
print(f'AUC-ROC: {auc_roc:.4f} (looks great!)')
print(f'AUC-PR: {auc_pr:.4f} (more honest about minority class)')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: the ROC curve plots TPR vs FPR across all thresholds and shows the full range of precision-recall trade-offs, AUC-ROC summarises ranking quality in one number where 0.5 = random and 1.0 = perfect, and for imbalanced classes use the Precision-Recall curve and Average Precision instead. Next up we explore regression metrics including MAE, MSE, RMSE, and R-squared.
Frequently asked questions
Is the “ROC Curves and AUC-ROC” lesson free?
Yes — the full text of “ROC Curves and AUC-ROC” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “ROC Curves and AUC-ROC”?
Learners will plot the ROC curve by varying the decision threshold, compute AUC, and interpret it as the probability that a random positive outranks a random negative. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “ROC Curves and AUC-ROC” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.