0Pricing
Machine Learning Academy · Lesson

Bias-Variance Trade-off: Underfitting vs Overfitting

Learners will plot training vs validation error curves, identify under- and overfitting regions, and understand the bias-variance decomposition conceptually.

Bias-Variance Trade-off: Underfitting vs Overfitting is a free Machine Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Central Tension in ML

Every machine learning model faces a fundamental tension between two competing forces: its ability to capture complex patterns in data (flexibility) and its ability to generalise those patterns to new examples (regularity). Too much flexibility leads to overfitting; too little leads to underfitting. Finding the sweet spot is the core challenge of model selection and hyperparameter tuning.

The bias-variance trade-off gives this tension a mathematical name and framework. Understanding it is essential for diagnosing what is wrong with a model and knowing exactly how to fix it.

Bias: Systematic Error from Wrong Assumptions

Bias is the error that comes from wrong assumptions in the learning algorithm. A high-bias model is too simple to capture the true relationship between features and the target — it makes systematically wrong predictions regardless of how much training data you give it.

The classic example: trying to fit a straight line to data that has a clear non-linear (curved) pattern. No matter how much data you have, the line will miss the curve. The model is underfitting — it has high bias because its linearity assumption is wrong for this problem.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Non-linear data: y = sin(x) + noise
np.random.seed(42)
X = np.sort(np.random.uniform(0, 10, 200)).reshape(-1, 1)
y = np.sin(X.ravel()) + np.random.randn(200) * 0.2

# High-bias model: straight line on non-linear data
linear_model = LinearRegression()
linear_model.fit(X, y)
y_pred_linear = linear_model.predict(X)

print(f'Linear model train MSE: {mean_squared_error(y, y_pred_linear):.3f}')
# High error even on training data -- high bias

Variance: Sensitivity to Training Data

Variance is the error that comes from sensitivity to small fluctuations in the training data. A high-variance model learns the training data so precisely — including its noise — that tiny changes in the training set produce very different models.

The classic example: a decision tree with unlimited depth that memorises every training example. On a different random training set drawn from the same distribution, it would learn a completely different tree. High variance means the model is overfitting — it captures noise rather than signal, and generalises poorly to new data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_classification(n_samples=200, n_features=5, random_state=42)

# Demonstrate high variance: train on two different splits
for seed in [1, 2, 3]:
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=seed)
    deep_tree = DecisionTreeClassifier()  # unlimited depth
    deep_tree.fit(X_train, y_train)
    print(f'seed={seed}: train={deep_tree.score(X_train, y_train):.2f} test={deep_tree.score(X_test, y_test):.2f}')
# Large variation in test scores across seeds = high variance

The Bias-Variance Decomposition

The expected test error of any model can be mathematically decomposed into three terms:

Expected Error = Bias² + Variance + Irreducible Noise

  • Bias²: squared systematic error from wrong assumptions. Reducible by using a more flexible model.
  • Variance: variability from sensitivity to training data. Reducible by using regularisation, more data, or simpler models.
  • Irreducible Noise: inherent randomness in the data that no model can eliminate.

You can only control bias and variance. The goal is to find the model complexity that minimises their sum.

Underfitting: Symptoms and Causes

Underfitting occurs when a model is too simple to capture the patterns in the data. Both training and test performance are poor. The model has high bias.

Symptoms of underfitting:

  • Training accuracy is low (the model cannot even fit training data well).
  • Training and validation errors are both high and close to each other (no large gap).
  • Adding more training data does not help significantly.

Causes: too simple an algorithm (e.g., linear model for non-linear data), too strong regularisation, too few features, or too few model parameters.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Underfitting: depth=1 is too shallow for 30-feature dataset
shallow = DecisionTreeClassifier(max_depth=1)
shallow.fit(X_train, y_train)
train_acc = shallow.score(X_train, y_train)
test_acc = shallow.score(X_test, y_test)

print(f'depth=1 (underfitting): train={train_acc:.3f} test={test_acc:.3f}')
print('Both low -- classic underfitting signature')

Overfitting: Symptoms and Causes

Overfitting occurs when a model learns training data too precisely — including noise — and fails to generalise. Training performance is very high but test performance is much lower. The model has high variance.

Symptoms of overfitting:

  • Training accuracy is very high (near 100%) while test accuracy is substantially lower.
  • Large gap between training and validation error curves.
  • Adding more training data consistently improves test performance.

Causes: too complex model, too few training examples, too many features (curse of dimensionality), or insufficient regularisation.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting: unlimited depth memorises training data
deep = DecisionTreeClassifier()  # max_depth=None
deep.fit(X_train, y_train)
train_acc = deep.score(X_train, y_train)
test_acc = deep.score(X_test, y_test)

print(f'depth=None (overfitting): train={train_acc:.3f} test={test_acc:.3f}')
print(f'Overfit gap: {train_acc - test_acc:.3f} -- classic overfitting signature')

Plotting the Bias-Variance Curve

The most illuminating visualisation is plotting training and validation error as a function of model complexity (e.g., tree depth). As complexity increases:

  • Training error monotonically decreases (or stays low).
  • Validation error forms a U-shape: initially high (underfitting), then falls to a minimum at the optimal complexity, then rises again (overfitting).

The optimal model complexity is at the bottom of the validation error U-curve.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

train_errors, test_errors = [], []
depths = range(1, 20)
for d in depths:
    m = DecisionTreeClassifier(max_depth=d).fit(X_train, y_train)
    train_errors.append(1 - m.score(X_train, y_train))
    test_errors.append(1 - m.score(X_test, y_test))

plt.plot(depths, train_errors, label='Train Error')
plt.plot(depths, test_errors, label='Test Error')
plt.xlabel('Tree Depth (Model Complexity)')
plt.ylabel('Classification Error')
plt.legend()
plt.title('Bias-Variance Trade-off Curve')
plt.show()

Learning Curves: Diagnosing with Data Size

A learning curve plots training and validation error as a function of training set size (rather than model complexity). Learning curves diagnose whether your model will benefit from more data:

  • If both curves converge to the same high error: high bias — more data will not help; you need a better model.
  • If training error is low but validation error is high, and the gap shrinks as data increases: high variance — more data will help; continue collecting.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

train_sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=None),  # overfitting model
    X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
)

plt.plot(train_sizes, train_scores.mean(axis=1), label='Train')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Validation')
plt.fill_between(train_sizes,
    val_scores.mean(axis=1) - val_scores.std(axis=1),
    val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.2)
plt.xlabel('Training Set Size')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Learning Curve')
plt.show()

Regularisation: Controlling Variance

Regularisation is the primary technique for reducing variance (overfitting) without changing the fundamental model family. It adds a penalty for model complexity to the loss function, discouraging the model from fitting noise.

Common regularisation techniques:

  • Tree depth limit (max_depth): prevents trees from growing deep enough to memorise training data.
  • L2 penalty (Ridge): shrinks weights towards zero.
  • L1 penalty (Lasso): drives some weights to exactly zero (feature selection).
  • Dropout (neural networks): randomly zeroes activations during training.
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

# Compare regularised vs non-regularised tree
for max_depth in [None, 10, 5, 3, 2]:
    model = DecisionTreeClassifier(max_depth=max_depth, random_state=42)
    cv = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f'max_depth={max_depth}: CV acc = {cv.mean():.3f} (+/- {cv.std():.3f})')

The Sweet Spot: Optimal Model Complexity

The optimal model complexity lies at the minimum of the test/validation error curve. To find it systematically:

  1. Define a range of complexity values (e.g., tree depth 1 to 20, or regularisation parameter C from 0.001 to 100).
  2. For each value, train on a training fold and evaluate on a validation fold (use cross-validation for stability).
  3. Select the complexity that gives the best cross-validated score.
  4. Retrain the final model on all training data with that optimal complexity.

This process is called hyperparameter tuning, and you will automate it with GridSearchCV in a later lesson.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

best_score, best_depth = 0, 1
for depth in range(1, 25):
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)
    score = cross_val_score(model, X, y, cv=5).mean()
    if score > best_score:
        best_score, best_depth = score, depth
    print(f'depth={depth:2d}: CV acc = {score:.3f}')

print(f'\nOptimal depth: {best_depth} with CV acc: {best_score:.3f}')

Practical Bias-Variance Cheat Sheet

A quick-reference guide for diagnosing and fixing bias-variance problems:

  • High bias (underfitting) fixes: use a more complex model, add more features, reduce regularisation strength, create polynomial or interaction features.
  • High variance (overfitting) fixes: collect more training data, apply stronger regularisation, reduce model complexity, use ensembles (bagging), apply dropout (neural nets), perform feature selection to remove noise features.
  • Unclear which problem: plot learning curves — if train and val errors both high → bias; if large gap → variance.

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: bias is systematic error from overly simple models that underfit the data, variance is sensitivity to training data that causes overfitting and poor generalisation, and the optimal model complexity minimises the sum of bias and variance, found by examining the U-shaped validation error curve or using learning curves to distinguish the two problems. Next up we build a DummyClassifier baseline and establish the minimum performance threshold that every real model must beat.

Frequently asked questions

Is the “Bias-Variance Trade-off: Underfitting vs Overfitting” lesson free?

Yes — the full text of “Bias-Variance Trade-off: Underfitting vs Overfitting” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Bias-Variance Trade-off: Underfitting vs Overfitting”?

Learners will plot training vs validation error curves, identify under- and overfitting regions, and understand the bias-variance decomposition conceptually. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Bias-Variance Trade-off: Underfitting vs Overfitting” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Why You Cannot Evaluate on Training Data
  2. train_test_split: Ratios, Seeds, and Stratification
  3. Bias-Variance Trade-off: Underfitting vs Overfitting
  4. Baseline Models: Always Beat the Dummy Classifier
← Back to Machine Learning Academy