0Pricing
Machine Learning Academy · Pelajaran

Kurva ROC dan AUC-ROC

Peserta didik akan menggambar kurva ROC dengan mengubah ambang keputusan, menghitung AUC, dan menafsirkannya sebagai probabilitas bahwa positif acak memiliki peringkat lebih tinggi daripada negatif acak.

Kurva ROC dan AUC-ROC adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 2 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.

Limitations of Fixed-Threshold Metrics

Precision, recall, and F1-score all depend on a fixed decision threshold (typically 0.5 for binary classifiers). But the optimal threshold varies by application: a spam filter might use 0.9 (strict, to avoid false positives), while a fraud detector might use 0.1 (lenient, to avoid missing fraud). Evaluating a model at only one threshold gives an incomplete picture of its capability. The ROC curve (Receiver Operating Characteristic) solves this by showing performance across all possible thresholds simultaneously, letting you understand the full range of precision-recall trade-offs the model can achieve.

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

for t in [0.1, 0.3, 0.5, 0.7, 0.9]:
    y_pred = (proba >= t).astype(int)
    print(f'thresh={t}: prec={precision_score(y_te,y_pred,zero_division=0):.3f}, '
          f'rec={recall_score(y_te,y_pred,zero_division=0):.3f}')

True Positive Rate and False Positive Rate

The ROC curve is built from two metrics computed at each threshold: True Positive Rate (TPR) (same as recall/sensitivity): TPR = TP / (TP + FN) — what fraction of actual positives did we catch? False Positive Rate (FPR): FPR = FP / (FP + TN) — what fraction of actual negatives did we incorrectly flag? As the threshold decreases, both TPR and FPR increase. The ROC curve plots TPR vs FPR as the threshold varies from 1 (predict nothing positive) to 0 (predict everything positive). A perfect classifier reaches (0, 1) — zero false alarms, all true positives found.

import numpy as np
from sklearn.metrics import confusion_matrix

def tpr_fpr(y_true, y_pred):
    TN, FP, FN, TP = confusion_matrix(y_true, y_pred).ravel()
    tpr = TP / (TP + FN)   # Recall = sensitivity
    fpr = FP / (FP + TN)   # False positive rate = 1 - specificity
    return tpr, fpr

# Example predictions at different thresholds
y_true = np.array([1,1,1,0,0,0,0,0,0,0])

for thresh in [0.3, 0.5, 0.7]:
    # Simulated probabilities
    proba = np.array([0.9, 0.8, 0.4, 0.6, 0.5, 0.3, 0.2, 0.1, 0.05, 0.01])
    y_pred = (proba >= thresh).astype(int)
    tpr, fpr = tpr_fpr(y_true, y_pred)
    print(f'thresh={thresh}: TPR={tpr:.2f}, FPR={fpr:.2f}')

Plotting the ROC Curve

Scikit-learn's roc_curve() computes the (FPR, TPR) pairs across all thresholds in a single call. RocCurveDisplay creates a publication-ready plot. The ROC curve for a random classifier is the diagonal line from (0,0) to (1,1) — it achieves no better than chance at any threshold. A good classifier bows toward the upper-left corner, achieving high TPR with low FPR. The further the curve is from the diagonal, the better the classifier. The area under this curve (AUC) summarises performance in a single number.

from sklearn.metrics import roc_curve, RocCurveDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)

RocCurveDisplay.from_estimator(clf, X_te, y_te)
plt.plot([0,1],[0,1],'k--', label='Random classifier')
plt.title('ROC Curve')
plt.legend()
plt.show()

AUC-ROC: Area Under the Curve

AUC-ROC (Area Under the ROC Curve) summarises the ROC curve into a single number between 0 and 1. AUC = 0.5: no better than random (diagonal line). AUC = 1.0: perfect classifier (no false positives at any recall level). AUC = 0.0: perfect inverse classifier — every prediction is wrong (useful if you flip predictions). An AUC of 0.8 means that 80% of the time, a randomly selected positive sample receives a higher predicted probability than a randomly selected negative sample. AUC is threshold-independent — it evaluates the ranking quality of the probability scores, not a specific prediction decision.

from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

auc = roc_auc_score(y_te, proba)
print(f'AUC-ROC: {auc:.4f}')

# Interpretation: AUC = P(score(pos) > score(neg))
import numpy as np
pos_scores = proba[y_te == 1]
neg_scores = proba[y_te == 0]
mc_auc = np.mean([p > n for p in pos_scores for n in neg_scores])
print(f'Monte Carlo AUC: {mc_auc:.4f}')  # Should match roc_auc_score

Interpreting AUC Values in Practice

Guidelines for interpreting AUC: 0.5 = random classifier, useless; 0.6-0.7 = poor, but better than chance; 0.7-0.8 = fair — acceptable for some applications; 0.8-0.9 = good — solid production model; 0.9-1.0 = excellent — likely near state-of-the-art for the problem. Beware of AUC inflation: temporal data leakage (using future information), label leakage, or highly imbalanced classes can artificially inflate AUC. Always sanity-check with calibration curves and by inspecting predictions on specific examples.

# AUC interpretation guide
auc_guide = [
    (0.50, 0.60, 'Random to poor'),
    (0.60, 0.70, 'Poor but predictive'),
    (0.70, 0.80, 'Fair'),
    (0.80, 0.90, 'Good'),
    (0.90, 1.00, 'Excellent'),
]

for low, high, label in auc_guide:
    print(f'AUC [{low:.2f}, {high:.2f}): {label}')

# Note: domain matters -- AUC 0.75 might be excellent
# for predicting disease 10 years in advance but poor
# for predicting email spam from clear signals

Comparing Multiple Models on the ROC Curve

Overlaying multiple ROC curves on the same plot enables direct model comparison. Each model's curve and AUC value appear together. A model is better if its ROC curve is uniformly above another's at every FPR. If curves cross, neither model dominates — one is better at low false positive rates, the other at high false positive rates, and the business context determines which is preferable. AUC comparison is a quick summary, but always check curves at the operating point you actually intend to use.

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

models = [
    ('Logistic Reg', LogisticRegression(max_iter=1000)),
    ('Decision Tree', DecisionTreeClassifier(max_depth=5)),
    ('KNN', KNeighborsClassifier(n_neighbors=5)),
]

for name, model in models:
    model.fit(StandardScaler().fit_transform(X_tr) if name!='Decision Tree' else X_tr, y_tr)
    proba = model.predict_proba(X_te)[:, 1]
    fpr, tpr, _ = roc_curve(y_te, proba)
    auc = roc_auc_score(y_te, proba)
    plt.plot(fpr, tpr, label=f'{name} (AUC={auc:.3f})')

plt.plot([0,1],[0,1],'k--')
plt.legend()
plt.show()

The Precision-Recall Curve: Better for Imbalanced Data

For highly imbalanced datasets, the ROC curve can be misleadingly optimistic. When there are very few positives (e.g., 1% fraud rate), even a model with many false positives achieves a low FPR because the denominator (TN+FP) is huge. The Precision-Recall curve plots precision vs recall at different thresholds and is not affected by class imbalance. The area under the PR curve (Average Precision, AP) is a better summary metric for imbalanced problems. Use ROC for balanced classes; use Precision-Recall curve for imbalanced classes.

from sklearn.metrics import PrecisionRecallDisplay, average_precision_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = LogisticRegression(max_iter=10000)
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

ap = average_precision_score(y_te, proba)
print(f'Average Precision (AP): {ap:.4f}')

PrecisionRecallDisplay.from_estimator(clf, X_te, y_te)
plt.title(f'Precision-Recall Curve (AP={ap:.3f})')
plt.show()

AUC in Cross-Validation

To get a reliable AUC estimate that is not biased by a single train-test split, use cross_val_score with scoring='roc_auc'. This computes AUC on each fold's validation set and returns all fold scores. The mean is the best estimate; the standard deviation shows stability. High mean AUC with low std indicates a robust model. High AUC with high std suggests the model is sensitive to which samples appear in each fold — consider whether the data has natural groupings (e.g., multiple samples per patient) that should be kept together in folds.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ('sc', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000))
])

aucs = cross_val_score(pipe, X, y, cv=10, scoring='roc_auc')
print(f'AUC per fold: {aucs.round(3)}')
print(f'Mean AUC: {aucs.mean():.4f}')
print(f'Std AUC:  {aucs.std():.4f}')

Setting an Operating Point on the ROC Curve

After evaluating a model with the ROC curve, you must choose a specific operating point (a threshold) for production use. Three strategies: (1) Youden's J statistic: choose the threshold that maximises (TPR - FPR) — the point farthest above the diagonal. (2) Business constraint: set maximum allowable FPR (e.g., flag at most 5% of legitimate transactions as fraud) and find the corresponding threshold. (3) Cost optimisation: if you know the cost of FP and FN, choose the threshold that minimises total cost. Document the chosen threshold along with the model for reproducible deployment.

from sklearn.metrics import roc_curve
import numpy as np

fpr, tpr, thresholds = roc_curve(y_te, proba)

# Strategy 1: Youden's J (max TPR - FPR)
youden_j = tpr - fpr
best_idx = np.argmax(youden_j)
best_threshold = thresholds[best_idx]
print(f'Youden threshold: {best_threshold:.3f}')
print(f'TPR={tpr[best_idx]:.3f}, FPR={fpr[best_idx]:.3f}')

# Strategy 2: Max TPR with FPR <= 0.05
allowed_fpr = 0.05
mask = fpr <= allowed_fpr
idx = np.where(mask)[0][-1]  # Highest TPR satisfying FPR constraint
print(f'\nAt FPR<={allowed_fpr}: TPR={tpr[idx]:.3f}, threshold={thresholds[idx]:.3f}')

Multi-Class ROC-AUC

For multi-class classification, ROC-AUC is extended by computing AUC for each class in a One-vs-Rest (OvR) setup: treat class k as positive and all others as negative, compute AUC, then average across classes. Scikit-learn's roc_auc_score supports this with multi_class='ovr' and average='macro' or average='weighted'. This requires probability estimates from predict_proba(), so the classifier must support probability output. Macro AUC treats all classes equally; weighted AUC accounts for class frequency.

from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)

clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_tr, y_tr)
proba_all = clf.predict_proba(X_te)  # Shape: (n_samples, n_classes)

auc_macro    = roc_auc_score(y_te, proba_all, multi_class='ovr', average='macro')
auc_weighted = roc_auc_score(y_te, proba_all, multi_class='ovr', average='weighted')

print(f'Macro AUC (OvR):    {auc_macro:.4f}')
print(f'Weighted AUC (OvR): {auc_weighted:.4f}')

AUC for Imbalanced Datasets: A Warning

AUC-ROC can be misleadingly optimistic for severely imbalanced datasets. When only 0.1% of transactions are fraud, even a model that captures 80% of fraud cases while flagging 5% of legitimate ones achieves an AUC near 0.9 — but in absolute terms, flagging 5% of millions of legitimate transactions creates enormous operational cost. In such cases, AUC-PR (area under the Precision-Recall curve) is more informative because it focuses only on the positive class performance and is not inflated by the large number of true negatives. For extreme imbalance (<1% positive rate), always report AUC-PR alongside AUC-ROC.

from sklearn.metrics import roc_auc_score, average_precision_score
from sklearn.linear_model import LogisticRegression
import numpy as np

np.random.seed(42)
# Extremely imbalanced: 1% positive rate
n_neg, n_pos = 9900, 100
X = np.vstack([np.random.randn(n_neg, 5),
               np.random.randn(n_pos, 5) + 1.5])
y = np.array([0]*n_neg + [1]*n_pos)

clf = LogisticRegression(class_weight='balanced', max_iter=1000)
clf.fit(X, y)
proba = clf.predict_proba(X)[:, 1]

auc_roc = roc_auc_score(y, proba)
auc_pr  = average_precision_score(y, proba)

print(f'AUC-ROC: {auc_roc:.4f} (looks great!)')
print(f'AUC-PR:  {auc_pr:.4f} (more honest about minority class)')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the ROC curve plots TPR vs FPR across all thresholds and shows the full range of precision-recall trade-offs, AUC-ROC summarises ranking quality in one number where 0.5 = random and 1.0 = perfect, and for imbalanced classes use the Precision-Recall curve and Average Precision instead. Next up we explore regression metrics including MAE, MSE, RMSE, and R-squared.

Pertanyaan yang Sering Diajukan

Apakah pelajaran “Kurva ROC dan AUC-ROC” gratis?

Ya — teks lengkap “Kurva ROC dan AUC-ROC” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Apa yang akan aku pelajari di “Kurva ROC dan AUC-ROC”?

Peserta didik akan menggambar kurva ROC dengan mengubah ambang keputusan, menghitung AUC, dan menafsirkannya sebagai probabilitas bahwa positif acak memiliki peringkat lebih tinggi daripada negatif a… Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.

Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?

Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 2 dari 4.

Berapa lama pelajaran “Kurva ROC dan AUC-ROC” memakan waktu?

Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.

Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?

Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.

Semua pelajaran dalam kursus ini

  1. Metrik Klasifikasi: Akurasi, Presisi, Recall, F1
  2. Kurva ROC dan AUC-ROC
  3. Metrik Regresi: MAE, MSE, RMSE, dan R-Kuadrat
  4. Memilih Metrik yang Tepat untuk Masalah Bisnis
← Kembali ke Machine Learning Academy