0Pricing
Machine Learning Academy · Lektion

Bias-Variance-Trade-off: Underfitting vs. Overfitting

Zeichnen Sie Kurven für Trainings- und Validierungsfehler, identifizieren Sie Bereiche mit Underfitting und Overfitting und verstehen Sie das Konzept der Bias-Variance-Zerlegung.

Bias-Variance-Trade-off: Underfitting vs. Overfitting ist eine kostenlose Machine Learning Academy-Lektion auf CoddyKit. Dies ist Lektion 3 von 4. Du kannst die komplette Lektion unten kostenlos lesen – dann übst du sie direkt im Browser mit einem integrierten Code-Editor und einem KI-Tutor rund um die Uhr. Sie ist Teil des Machine Learning Academy-Lernpfads, und dein Fortschritt wird über Web und CoddyKit-App synchronisiert. Der Machine Learning Academy-Kurs umfasst insgesamt 4 Lektionen.

Teile dieser Lektion wurden noch nicht übersetzt und werden auf Englisch angezeigt.

The Central Tension in ML

Every machine learning model faces a fundamental tension between two competing forces: its ability to capture complex patterns in data (flexibility) and its ability to generalise those patterns to new examples (regularity). Too much flexibility leads to overfitting; too little leads to underfitting. Finding the sweet spot is the core challenge of model selection and hyperparameter tuning.

The bias-variance trade-off gives this tension a mathematical name and framework. Understanding it is essential for diagnosing what is wrong with a model and knowing exactly how to fix it.

Bias: Systematic Error from Wrong Assumptions

Bias is the error that comes from wrong assumptions in the learning algorithm. A high-bias model is too simple to capture the true relationship between features and the target — it makes systematically wrong predictions regardless of how much training data you give it.

The classic example: trying to fit a straight line to data that has a clear non-linear (curved) pattern. No matter how much data you have, the line will miss the curve. The model is underfitting — it has high bias because its linearity assumption is wrong for this problem.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Non-linear data: y = sin(x) + noise
np.random.seed(42)
X = np.sort(np.random.uniform(0, 10, 200)).reshape(-1, 1)
y = np.sin(X.ravel()) + np.random.randn(200) * 0.2

# High-bias model: straight line on non-linear data
linear_model = LinearRegression()
linear_model.fit(X, y)
y_pred_linear = linear_model.predict(X)

print(f'Linear model train MSE: {mean_squared_error(y, y_pred_linear):.3f}')
# High error even on training data -- high bias

Variance: Sensitivity to Training Data

Variance is the error that comes from sensitivity to small fluctuations in the training data. A high-variance model learns the training data so precisely — including its noise — that tiny changes in the training set produce very different models.

The classic example: a decision tree with unlimited depth that memorises every training example. On a different random training set drawn from the same distribution, it would learn a completely different tree. High variance means the model is overfitting — it captures noise rather than signal, and generalises poorly to new data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_classification(n_samples=200, n_features=5, random_state=42)

# Demonstrate high variance: train on two different splits
for seed in [1, 2, 3]:
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=seed)
    deep_tree = DecisionTreeClassifier()  # unlimited depth
    deep_tree.fit(X_train, y_train)
    print(f'seed={seed}: train={deep_tree.score(X_train, y_train):.2f} test={deep_tree.score(X_test, y_test):.2f}')
# Large variation in test scores across seeds = high variance

The Bias-Variance Decomposition

The expected test error of any model can be mathematically decomposed into three terms:

Expected Error = Bias² + Variance + Irreducible Noise

  • Bias²: squared systematic error from wrong assumptions. Reducible by using a more flexible model.
  • Variance: variability from sensitivity to training data. Reducible by using regularisation, more data, or simpler models.
  • Irreducible Noise: inherent randomness in the data that no model can eliminate.

You can only control bias and variance. The goal is to find the model complexity that minimises their sum.

Underfitting: Symptoms and Causes

Underfitting occurs when a model is too simple to capture the patterns in the data. Both training and test performance are poor. The model has high bias.

Symptoms of underfitting:

  • Training accuracy is low (the model cannot even fit training data well).
  • Training and validation errors are both high and close to each other (no large gap).
  • Adding more training data does not help significantly.

Causes: too simple an algorithm (e.g., linear model for non-linear data), too strong regularisation, too few features, or too few model parameters.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Underfitting: depth=1 is too shallow for 30-feature dataset
shallow = DecisionTreeClassifier(max_depth=1)
shallow.fit(X_train, y_train)
train_acc = shallow.score(X_train, y_train)
test_acc = shallow.score(X_test, y_test)

print(f'depth=1 (underfitting): train={train_acc:.3f} test={test_acc:.3f}')
print('Both low -- classic underfitting signature')

Overfitting: Symptoms and Causes

Overfitting occurs when a model learns training data too precisely — including noise — and fails to generalise. Training performance is very high but test performance is much lower. The model has high variance.

Symptoms of overfitting:

  • Training accuracy is very high (near 100%) while test accuracy is substantially lower.
  • Large gap between training and validation error curves.
  • Adding more training data consistently improves test performance.

Causes: too complex model, too few training examples, too many features (curse of dimensionality), or insufficient regularisation.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting: unlimited depth memorises training data
deep = DecisionTreeClassifier()  # max_depth=None
deep.fit(X_train, y_train)
train_acc = deep.score(X_train, y_train)
test_acc = deep.score(X_test, y_test)

print(f'depth=None (overfitting): train={train_acc:.3f} test={test_acc:.3f}')
print(f'Overfit gap: {train_acc - test_acc:.3f} -- classic overfitting signature')

Plotting the Bias-Variance Curve

The most illuminating visualisation is plotting training and validation error as a function of model complexity (e.g., tree depth). As complexity increases:

  • Training error monotonically decreases (or stays low).
  • Validation error forms a U-shape: initially high (underfitting), then falls to a minimum at the optimal complexity, then rises again (overfitting).

The optimal model complexity is at the bottom of the validation error U-curve.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

train_errors, test_errors = [], []
depths = range(1, 20)
for d in depths:
    m = DecisionTreeClassifier(max_depth=d).fit(X_train, y_train)
    train_errors.append(1 - m.score(X_train, y_train))
    test_errors.append(1 - m.score(X_test, y_test))

plt.plot(depths, train_errors, label='Train Error')
plt.plot(depths, test_errors, label='Test Error')
plt.xlabel('Tree Depth (Model Complexity)')
plt.ylabel('Classification Error')
plt.legend()
plt.title('Bias-Variance Trade-off Curve')
plt.show()

Learning Curves: Diagnosing with Data Size

A learning curve plots training and validation error as a function of training set size (rather than model complexity). Learning curves diagnose whether your model will benefit from more data:

  • If both curves converge to the same high error: high bias — more data will not help; you need a better model.
  • If training error is low but validation error is high, and the gap shrinks as data increases: high variance — more data will help; continue collecting.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

train_sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=None),  # overfitting model
    X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
)

plt.plot(train_sizes, train_scores.mean(axis=1), label='Train')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Validation')
plt.fill_between(train_sizes,
    val_scores.mean(axis=1) - val_scores.std(axis=1),
    val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.2)
plt.xlabel('Training Set Size')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Learning Curve')
plt.show()

Regularisation: Controlling Variance

Regularisation is the primary technique for reducing variance (overfitting) without changing the fundamental model family. It adds a penalty for model complexity to the loss function, discouraging the model from fitting noise.

Common regularisation techniques:

  • Tree depth limit (max_depth): prevents trees from growing deep enough to memorise training data.
  • L2 penalty (Ridge): shrinks weights towards zero.
  • L1 penalty (Lasso): drives some weights to exactly zero (feature selection).
  • Dropout (neural networks): randomly zeroes activations during training.
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

# Compare regularised vs non-regularised tree
for max_depth in [None, 10, 5, 3, 2]:
    model = DecisionTreeClassifier(max_depth=max_depth, random_state=42)
    cv = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f'max_depth={max_depth}: CV acc = {cv.mean():.3f} (+/- {cv.std():.3f})')

The Sweet Spot: Optimal Model Complexity

The optimal model complexity lies at the minimum of the test/validation error curve. To find it systematically:

  1. Define a range of complexity values (e.g., tree depth 1 to 20, or regularisation parameter C from 0.001 to 100).
  2. For each value, train on a training fold and evaluate on a validation fold (use cross-validation for stability).
  3. Select the complexity that gives the best cross-validated score.
  4. Retrain the final model on all training data with that optimal complexity.

This process is called hyperparameter tuning, and you will automate it with GridSearchCV in a later lesson.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

best_score, best_depth = 0, 1
for depth in range(1, 25):
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)
    score = cross_val_score(model, X, y, cv=5).mean()
    if score > best_score:
        best_score, best_depth = score, depth
    print(f'depth={depth:2d}: CV acc = {score:.3f}')

print(f'\nOptimal depth: {best_depth} with CV acc: {best_score:.3f}')

Practical Bias-Variance Cheat Sheet

A quick-reference guide for diagnosing and fixing bias-variance problems:

  • High bias (underfitting) fixes: use a more complex model, add more features, reduce regularisation strength, create polynomial or interaction features.
  • High variance (overfitting) fixes: collect more training data, apply stronger regularisation, reduce model complexity, use ensembles (bagging), apply dropout (neural nets), perform feature selection to remove noise features.
  • Unclear which problem: plot learning curves — if train and val errors both high → bias; if large gap → variance.

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: bias is systematic error from overly simple models that underfit the data, variance is sensitivity to training data that causes overfitting and poor generalisation, and the optimal model complexity minimises the sum of bias and variance, found by examining the U-shaped validation error curve or using learning curves to distinguish the two problems. Next up we build a DummyClassifier baseline and establish the minimum performance threshold that every real model must beat.

Häufig gestellte Fragen

Ist die Lektion „Bias-Variance-Trade-off: Underfitting vs. Overfitting“ kostenlos?

Ja — der vollständige Text von „Bias-Variance-Trade-off: Underfitting vs. Overfitting“ ist hier im Web kostenlos zu lesen. Um sie interaktiv zu üben (integrierter Code-Editor und 24/7 KI-Tutor) und den Rest des Machine Learning Academy-Kurses freizuschalten, upgrade auf CoddyKit PRO. Der Machine Learning Academy-Kurs umfasst insgesamt 4 Lektionen.

Was lerne ich in „Bias-Variance-Trade-off: Underfitting vs. Overfitting“?

Zeichnen Sie Kurven für Trainings- und Validierungsfehler, identifizieren Sie Bereiche mit Underfitting und Overfitting und verstehen Sie das Konzept der Bias-Variance-Zerlegung. Du übst Machine Learning Academy mit praktischem Code, den du direkt im Browser ausführst, und ein 24/7 KI-Tutor beantwortet deine Fragen während du die Lektion bearbeitest.

Brauche ich Erfahrung, um Machine Learning Academy zu starten?

Keine Vorkenntnisse erforderlich. Machine Learning Academy auf CoddyKit ist für Anfänger bis fortgeschrittene Lernende strukturiert, sodass du hier starten oder von Anfang an beginnen und in deinem eigenen Tempo voranschreiten kannst. Dies ist Lektion 3 von 4.

Wie lange dauert die Lektion „Bias-Variance-Trade-off: Underfitting vs. Overfitting“?

Die meisten CoddyKit-Lektionen dauern etwa 5–10 Minuten. Jede ist kompakt und interaktiv, sodass du stetig Fortschritte machst und genau dort weitermachst, wo du aufgehört hast – im Web und in der App.

Kann ich in dieser Machine Learning Academy-Lektion Code schreiben und ausführen?

Ja. Jede Machine Learning Academy-Lektion enthält einen integrierten Code-Editor, sodass du echten Code direkt in deinem Browser schreibst und ausführst und sofort KI-Feedback erhältst — ohne lokale Einrichtung erforderlich.

Alle Lektionen in diesem Kurs

  1. Warum Sie nicht mit Trainingsdaten evaluieren können
  2. train_test_split: Verhältnisse, Seeds und Stratifikation
  3. Bias-Variance-Trade-off: Underfitting vs. Overfitting
  4. Baseline-Modelle: Immer besser als der DummyClassifier
← Zurück zu Machine Learning Academy