0Pricing
Machine Learning Academy · Leçon

Compromis biais-variance : sous-ajustement ou surajustement

Tracez les courbes d’erreur d’entraînement et de validation, identifiez les zones de sous-ajustement et de surajustement et comprenez conceptuellement la décomposition biais-variance.

Compromis biais-variance : sous-ajustement ou surajustement est une leçon Machine Learning Academy gratuite sur CoddyKit. Ceci est la leçon 3 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Machine Learning Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Machine Learning Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

The Central Tension in ML

Every machine learning model faces a fundamental tension between two competing forces: its ability to capture complex patterns in data (flexibility) and its ability to generalise those patterns to new examples (regularity). Too much flexibility leads to overfitting; too little leads to underfitting. Finding the sweet spot is the core challenge of model selection and hyperparameter tuning.

The bias-variance trade-off gives this tension a mathematical name and framework. Understanding it is essential for diagnosing what is wrong with a model and knowing exactly how to fix it.

Bias: Systematic Error from Wrong Assumptions

Bias is the error that comes from wrong assumptions in the learning algorithm. A high-bias model is too simple to capture the true relationship between features and the target — it makes systematically wrong predictions regardless of how much training data you give it.

The classic example: trying to fit a straight line to data that has a clear non-linear (curved) pattern. No matter how much data you have, the line will miss the curve. The model is underfitting — it has high bias because its linearity assumption is wrong for this problem.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Non-linear data: y = sin(x) + noise
np.random.seed(42)
X = np.sort(np.random.uniform(0, 10, 200)).reshape(-1, 1)
y = np.sin(X.ravel()) + np.random.randn(200) * 0.2

# High-bias model: straight line on non-linear data
linear_model = LinearRegression()
linear_model.fit(X, y)
y_pred_linear = linear_model.predict(X)

print(f'Linear model train MSE: {mean_squared_error(y, y_pred_linear):.3f}')
# High error even on training data -- high bias

Variance: Sensitivity to Training Data

Variance is the error that comes from sensitivity to small fluctuations in the training data. A high-variance model learns the training data so precisely — including its noise — that tiny changes in the training set produce very different models.

The classic example: a decision tree with unlimited depth that memorises every training example. On a different random training set drawn from the same distribution, it would learn a completely different tree. High variance means the model is overfitting — it captures noise rather than signal, and generalises poorly to new data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_classification(n_samples=200, n_features=5, random_state=42)

# Demonstrate high variance: train on two different splits
for seed in [1, 2, 3]:
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=seed)
    deep_tree = DecisionTreeClassifier()  # unlimited depth
    deep_tree.fit(X_train, y_train)
    print(f'seed={seed}: train={deep_tree.score(X_train, y_train):.2f} test={deep_tree.score(X_test, y_test):.2f}')
# Large variation in test scores across seeds = high variance

The Bias-Variance Decomposition

The expected test error of any model can be mathematically decomposed into three terms:

Expected Error = Bias² + Variance + Irreducible Noise

  • Bias²: squared systematic error from wrong assumptions. Reducible by using a more flexible model.
  • Variance: variability from sensitivity to training data. Reducible by using regularisation, more data, or simpler models.
  • Irreducible Noise: inherent randomness in the data that no model can eliminate.

You can only control bias and variance. The goal is to find the model complexity that minimises their sum.

Underfitting: Symptoms and Causes

Underfitting occurs when a model is too simple to capture the patterns in the data. Both training and test performance are poor. The model has high bias.

Symptoms of underfitting:

  • Training accuracy is low (the model cannot even fit training data well).
  • Training and validation errors are both high and close to each other (no large gap).
  • Adding more training data does not help significantly.

Causes: too simple an algorithm (e.g., linear model for non-linear data), too strong regularisation, too few features, or too few model parameters.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Underfitting: depth=1 is too shallow for 30-feature dataset
shallow = DecisionTreeClassifier(max_depth=1)
shallow.fit(X_train, y_train)
train_acc = shallow.score(X_train, y_train)
test_acc = shallow.score(X_test, y_test)

print(f'depth=1 (underfitting): train={train_acc:.3f} test={test_acc:.3f}')
print('Both low -- classic underfitting signature')

Overfitting: Symptoms and Causes

Overfitting occurs when a model learns training data too precisely — including noise — and fails to generalise. Training performance is very high but test performance is much lower. The model has high variance.

Symptoms of overfitting:

  • Training accuracy is very high (near 100%) while test accuracy is substantially lower.
  • Large gap between training and validation error curves.
  • Adding more training data consistently improves test performance.

Causes: too complex model, too few training examples, too many features (curse of dimensionality), or insufficient regularisation.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting: unlimited depth memorises training data
deep = DecisionTreeClassifier()  # max_depth=None
deep.fit(X_train, y_train)
train_acc = deep.score(X_train, y_train)
test_acc = deep.score(X_test, y_test)

print(f'depth=None (overfitting): train={train_acc:.3f} test={test_acc:.3f}')
print(f'Overfit gap: {train_acc - test_acc:.3f} -- classic overfitting signature')

Plotting the Bias-Variance Curve

The most illuminating visualisation is plotting training and validation error as a function of model complexity (e.g., tree depth). As complexity increases:

  • Training error monotonically decreases (or stays low).
  • Validation error forms a U-shape: initially high (underfitting), then falls to a minimum at the optimal complexity, then rises again (overfitting).

The optimal model complexity is at the bottom of the validation error U-curve.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

train_errors, test_errors = [], []
depths = range(1, 20)
for d in depths:
    m = DecisionTreeClassifier(max_depth=d).fit(X_train, y_train)
    train_errors.append(1 - m.score(X_train, y_train))
    test_errors.append(1 - m.score(X_test, y_test))

plt.plot(depths, train_errors, label='Train Error')
plt.plot(depths, test_errors, label='Test Error')
plt.xlabel('Tree Depth (Model Complexity)')
plt.ylabel('Classification Error')
plt.legend()
plt.title('Bias-Variance Trade-off Curve')
plt.show()

Learning Curves: Diagnosing with Data Size

A learning curve plots training and validation error as a function of training set size (rather than model complexity). Learning curves diagnose whether your model will benefit from more data:

  • If both curves converge to the same high error: high bias — more data will not help; you need a better model.
  • If training error is low but validation error is high, and the gap shrinks as data increases: high variance — more data will help; continue collecting.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

train_sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=None),  # overfitting model
    X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
)

plt.plot(train_sizes, train_scores.mean(axis=1), label='Train')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Validation')
plt.fill_between(train_sizes,
    val_scores.mean(axis=1) - val_scores.std(axis=1),
    val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.2)
plt.xlabel('Training Set Size')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Learning Curve')
plt.show()

Regularisation: Controlling Variance

Regularisation is the primary technique for reducing variance (overfitting) without changing the fundamental model family. It adds a penalty for model complexity to the loss function, discouraging the model from fitting noise.

Common regularisation techniques:

  • Tree depth limit (max_depth): prevents trees from growing deep enough to memorise training data.
  • L2 penalty (Ridge): shrinks weights towards zero.
  • L1 penalty (Lasso): drives some weights to exactly zero (feature selection).
  • Dropout (neural networks): randomly zeroes activations during training.
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

# Compare regularised vs non-regularised tree
for max_depth in [None, 10, 5, 3, 2]:
    model = DecisionTreeClassifier(max_depth=max_depth, random_state=42)
    cv = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f'max_depth={max_depth}: CV acc = {cv.mean():.3f} (+/- {cv.std():.3f})')

The Sweet Spot: Optimal Model Complexity

The optimal model complexity lies at the minimum of the test/validation error curve. To find it systematically:

  1. Define a range of complexity values (e.g., tree depth 1 to 20, or regularisation parameter C from 0.001 to 100).
  2. For each value, train on a training fold and evaluate on a validation fold (use cross-validation for stability).
  3. Select the complexity that gives the best cross-validated score.
  4. Retrain the final model on all training data with that optimal complexity.

This process is called hyperparameter tuning, and you will automate it with GridSearchCV in a later lesson.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

best_score, best_depth = 0, 1
for depth in range(1, 25):
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)
    score = cross_val_score(model, X, y, cv=5).mean()
    if score > best_score:
        best_score, best_depth = score, depth
    print(f'depth={depth:2d}: CV acc = {score:.3f}')

print(f'\nOptimal depth: {best_depth} with CV acc: {best_score:.3f}')

Practical Bias-Variance Cheat Sheet

A quick-reference guide for diagnosing and fixing bias-variance problems:

  • High bias (underfitting) fixes: use a more complex model, add more features, reduce regularisation strength, create polynomial or interaction features.
  • High variance (overfitting) fixes: collect more training data, apply stronger regularisation, reduce model complexity, use ensembles (bagging), apply dropout (neural nets), perform feature selection to remove noise features.
  • Unclear which problem: plot learning curves — if train and val errors both high → bias; if large gap → variance.

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: bias is systematic error from overly simple models that underfit the data, variance is sensitivity to training data that causes overfitting and poor generalisation, and the optimal model complexity minimises the sum of bias and variance, found by examining the U-shaped validation error curve or using learning curves to distinguish the two problems. Next up we build a DummyClassifier baseline and establish the minimum performance threshold that every real model must beat.

Questions Fréquemment Posées

La leçon « Compromis biais-variance : sous-ajustement ou surajustement » est-elle gratuite ?

Oui — le texte complet de « Compromis biais-variance : sous-ajustement ou surajustement » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Machine Learning Academy, passe à CoddyKit PRO. Le cours Machine Learning Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Compromis biais-variance : sous-ajustement ou surajustement » ?

Tracez les courbes d’erreur d’entraînement et de validation, identifiez les zones de sous-ajustement et de surajustement et comprenez conceptuellement la décomposition biais-variance. Tu pratiques Machine Learning Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer Machine Learning Academy ?

Aucune expérience préalable n'est requise. Machine Learning Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 3 sur 4.

Combien de temps prend la leçon « Compromis biais-variance : sous-ajustement ou surajustement » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon Machine Learning Academy ?

Oui. Chaque leçon Machine Learning Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

  1. Pourquoi évaluer sur les données d’entraînement est impossible
  2. train_test_split : proportions, graines et stratification
  3. Compromis biais-variance : sous-ajustement ou surajustement
  4. Modèles de référence : toujours dépasser DummyClassifier
← Retour à Machine Learning Academy