0Pricing
Python Academy · Lesson

Model Evaluation and Cross-Validation

Evaluate models with accuracy, F1, ROC-AUC, and k-fold CV.

Model Evaluation and Cross-Validation is a free Python Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Python Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Train/Validation/Test Split

Use a three-way split: train to fit, validation to tune hyperparameters, test to report final performance. Never use the test set during development.

from sklearn.model_selection import train_test_split
import numpy as np

X = np.random.rand(1000, 10)
y = (X[:,0] > 0.5).astype(int)

X_tr, X_rest, y_tr, y_rest = train_test_split(X, y, test_size=0.3)
X_val, X_te, y_val, y_te = train_test_split(X_rest, y_rest, test_size=0.5)

Accuracy Score

accuracy_score is the fraction of correct predictions. Use it for balanced classes; for imbalanced data, prefer F1 or AUC.

from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
model = LogisticRegression().fit(X_tr, y_tr)
print("Accuracy:", accuracy_score(y_te, model.predict(X_te)))

Precision, Recall, F1

For imbalanced classes: precision = TP/(TP+FP), recall = TP/(TP+FN), F1 = harmonic mean of precision and recall.

from sklearn.metrics import precision_score, recall_score, f1_score, classification_report

# Use classification_report for a full summary:
print(classification_report(y_te, model.predict(X_te)))
# shows precision, recall, f1 for each class

Confusion Matrix

A confusion matrix shows true positives, false positives, true negatives, false negatives.

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

cm = confusion_matrix(y_te, model.predict(X_te))
print(cm)
# [[TN FP]
#  [FN TP]]

ROC Curve and AUC

roc_auc_score measures the area under the ROC curve. AUC = 1.0 is perfect; 0.5 = random.

from sklearn.metrics import roc_auc_score, roc_curve

y_prob = model.predict_proba(X_te)[:,1]
print("AUC:", roc_auc_score(y_te, y_prob))

fpr, tpr, thresholds = roc_curve(y_te, y_prob)
# plot fpr vs tpr for the full ROC curve

Regression Metrics

Use mean_squared_error (MSE), root_mean_squared_error (RMSE), and r2_score for regression tasks.

from sklearn.metrics import mean_squared_error, r2_score

y_pred = regression_model.predict(X_te)
mse = mean_squared_error(y_te, y_pred)
print("RMSE:", mse**0.5)
print("R²:", r2_score(y_te, y_pred))

k-Fold Cross-Validation

cross_val_score splits data into k folds, trains on k-1, evaluates on 1, and repeats k times. More robust than a single split.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(random_state=0)
scores = cross_val_score(LogisticRegression(), X, y, cv=10, scoring="accuracy")
print(f"{scores.mean():.3f} ± {scores.std():.3f}")

StratifiedKFold

Use StratifiedKFold to ensure each fold has the same class distribution — crucial for imbalanced datasets.

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(weights=[0.9,0.1], random_state=0)
cv = StratifiedKFold(n_splits=5)
scores = cross_val_score(LogisticRegression(), X, y, cv=cv, scoring="f1")
print("Stratified F1:", scores.mean())

Bias-Variance Trade-off

High bias = underfitting (model too simple). High variance = overfitting (model too complex). Cross-validation reveals which problem you have.

# Underfitting: low train score AND low val score
# → increase model complexity, add features

# Overfitting: high train score, low val score
# → regularise, reduce complexity, get more data

from sklearn.model_selection import validation_curve
import numpy as np

train_sc, val_sc = validation_curve(DecisionTreeClassifier(), X, y, param_name="max_depth", param_range=range(1,10))

Nested Cross-Validation

For unbiased hyperparameter tuning AND evaluation, use nested CV: inner loop tunes, outer loop evaluates.

from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import make_classification

X, y = make_classification(random_state=0)
gs = GridSearchCV(SVC(), {"C":[0.1,1,10]}, cv=5)
outer_scores = cross_val_score(gs, X, y, cv=5)
print("Nested CV accuracy:", outer_scores.mean())

Calibration

A classifier's probability estimates may be poorly calibrated. Use CalibratedClassifierCV for better probability estimates.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y)

cal = CalibratedClassifierCV(SVC(), cv=5).fit(X_tr, y_tr)
print(cal.predict_proba(X_te[:3]))

Quick Check

Why is cross-validation preferred over a single train/test split for evaluating models?

Recap

Use accuracy for balanced problems; F1/AUC for imbalanced ones. Evaluate with k-fold cross_val_score, use StratifiedKFold for imbalanced data, and nested CV when tuning hyperparameters to get an unbiased final estimate. Never peek at the test set during development.

Frequently asked questions

Is the “Model Evaluation and Cross-Validation” lesson free?

Yes — the full text of “Model Evaluation and Cross-Validation” is free to read here on the web, and the Python Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Python Academy course, upgrade to CoddyKit PRO.

What will I learn in “Model Evaluation and Cross-Validation”?

Evaluate models with accuracy, F1, ROC-AUC, and k-fold CV. You practise Python Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Python Academy?

No prior experience is required. Python Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Model Evaluation and Cross-Validation” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Python Academy lesson?

Yes. Every Python Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. The scikit-learn API: fit, transform, predict
  2. Linear Models: Regression and Classification
  3. Tree-Based Models: Decision Trees and Random Forests
  4. Model Evaluation and Cross-Validation
← Back to Python Academy