0Pricing
Machine Learning Academy · Lezione

Controllare la profondità dell'albero per prevenire l'overfitting

Addestrerà alberi con diversi valori di max_depth, osserverà il compromesso tra overfitting e underfitting e sceglierà la profondità tramite il punteggio di validazione.

Controllare la profondità dell'albero per prevenire l'overfitting è una lezione Machine Learning Academy gratuita su CoddyKit. Questa è la lezione 3 di 4. Puoi leggere la lezione completa qui gratuitamente — poi esercitati direttamente nel browser con un editor di codice integrato e un tutor IA disponibile 24/7. Fa parte del percorso di apprendimento Machine Learning Academy, e i tuoi progressi si sincronizzano tra il web e l'app CoddyKit. Il corso Machine Learning Academy include 4 lezioni in totale.

Parti di questa lezione non sono ancora state tradotte e vengono mostrate in inglese.

How Depth Leads to Overfitting

An unconstrained decision tree will grow until every training sample has its own leaf — achieving 100% training accuracy by memorising all data points, including noise. This is the extreme case of overfitting: the tree learns the idiosyncrasies of training data rather than general patterns. On any new data, such a tree performs poorly because its rules are too specific. Controlling tree depth is the primary regularisation mechanism for decision trees, analogous to choosing alpha in regularised linear models or choosing k in KNN.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Unlimited depth: memorises training data
tree = DecisionTreeClassifier(random_state=42)  # no max_depth
tree.fit(X_tr, y_tr)
print('Unlimited depth tree:')
print('  Tree depth:', tree.get_depth())
print('  Train accuracy:', tree.score(X_tr, y_tr).round(3))  # 1.000
print('  Test  accuracy:', tree.score(X_te, y_te).round(3))  # < 1.000

The max_depth Parameter

max_depth limits how many levels the tree can grow. With max_depth=1, the tree makes exactly one decision (a 'stump'). With max_depth=3, the tree can make up to three sequential questions. Shallower trees generalise better but may underfit; deeper trees fit training data better but risk overfitting. The right max_depth is a hyperparameter found through cross-validation. A useful heuristic: start around max_depth=3-5 and tune from there using a validation curve.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

for depth in [1, 2, 3, 5, 10, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_tr, y_tr)
    print(f'max_depth={str(depth):4}: train={tree.score(X_tr,y_tr):.3f}, '
          f'test={tree.score(X_te,y_te):.3f}')

Validation Curve for max_depth

Plotting training and validation accuracy vs max_depth reveals the classic bias-variance pattern. At depth 1, both training and validation accuracy are low (underfitting — high bias). As depth increases, training accuracy rises quickly to 100% while validation accuracy peaks and then decreases (overfitting — high variance). The optimal depth is where the validation curve peaks — before the train-test gap widens. Use cross-validation rather than a single validation split for a more reliable estimate of the peak location.

from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)

train_sc, val_sc = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y, param_name='max_depth',
    param_range=range(1, 16), cv=10
)

plt.plot(range(1,16), train_sc.mean(axis=1), label='Train')
plt.plot(range(1,16), val_sc.mean(axis=1), label='Validation')
plt.xlabel('max_depth'); plt.ylabel('Accuracy')
plt.title('Bias-Variance via max_depth')
plt.legend(); plt.show()

best_depth = np.argmax(val_sc.mean(axis=1)) + 1
print('Best depth:', best_depth)

min_samples_split: Minimum Samples to Split a Node

min_samples_split prevents splitting a node if it contains fewer than a specified number of samples. The default is 2 (any node with 2 or more samples can be split). Increasing this value forces the tree to wait for more evidence before making a decision, which prevents highly specific splits on tiny groups that are likely noise. Setting min_samples_split=20 means no node with fewer than 20 samples will be split further. This is especially useful for datasets with many samples and many rare subgroups.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_split in [2, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_split=min_split,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_split={min_split:3}: depth={tree.get_depth()}, CV acc={score:.3f}')

min_samples_leaf: Minimum Samples in a Leaf

min_samples_leaf requires that after any split, each resulting child node must have at least this many samples. If a candidate split would create a child with too few samples, that split is rejected. This is a stronger constraint than min_samples_split because it guarantees a minimum number of samples in every leaf. Larger min_samples_leaf produces smaller, less deep trees with smoother decision boundaries. For probability estimation, this parameter is critical: leaf nodes with very few samples produce unreliable probability estimates.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_leaf in [1, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_leaf=min_leaf,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_leaf={min_leaf:3}: leaves={tree.get_n_leaves():4}, CV acc={score:.3f}')

max_leaf_nodes: Limiting Total Leaves

Instead of controlling depth, you can directly limit the maximum number of leaf nodes. With max_leaf_nodes=10, the tree grows until it has exactly 10 leaves, choosing the split that provides the greatest impurity reduction at each step (best-first growth, not depth-first). This produces more balanced trees than depth-limited growth, as splits are allocated where they are most informative rather than being forced to equal depth. Best-first growth can sometimes give better performance than depth-limited growth on the same number of leaves.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for n_leaves in [2, 4, 8, 16, 32, 64]:
    tree = DecisionTreeClassifier(
        max_leaf_nodes=n_leaves,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'max_leaf_nodes={n_leaves:3}: depth={tree.get_depth()}, '
          f'actual leaves={tree.get_n_leaves()}, CV acc={score:.3f}')

Cost-Complexity Pruning with ccp_alpha

Post-pruning reduces a fully-grown tree after training by removing splits that contribute little predictive value. Scikit-learn implements cost-complexity pruning via the ccp_alpha parameter. Larger ccp_alpha prunes more aggressively, resulting in smaller trees. You can find the optimal alpha by computing the pruning path with cost_complexity_pruning_path(), which returns the alpha values at which each subtree becomes optimal. Cross-validate over these alphas to find the one that maximises validation accuracy.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Get pruning path
tree = DecisionTreeClassifier(random_state=42)
path = tree.cost_complexity_pruning_path(X_tr, y_tr)
ccp_alphas = path.ccp_alphas

# Cross-validate each alpha
best_alpha, best_score = 0, 0
for alpha in ccp_alphas[::5]:  # sample every 5th
    t = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42)
    score = cross_val_score(t, X_tr, y_tr, cv=5).mean()
    if score > best_score:
        best_score, best_alpha = score, alpha

print(f'Best ccp_alpha: {best_alpha:.5f}, CV acc: {best_score:.3f}')

Visualising Overfitting vs Good Fit

Plotting a shallow tree vs a deep tree side-by-side on a 2D dataset makes the overfitting effect visually obvious. A shallow tree (depth=2) draws a few rectangular decision regions that capture the main structure of the data. A deep tree (depth=20) creates hundreds of tiny rectangular regions that perfectly trace the training points, including noise. Good generalisation comes from finding the depth where test accuracy is highest — this is always below the fully-grown tree depth, and finding it is the goal of every depth-tuning experiment.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
import numpy as np

X, y = make_moons(n_samples=300, noise=0.3, random_state=42)

for depth in [2, 5, 20]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X, y)
    train_acc = tree.score(X, y)
    print(f'depth={depth:2}: train={train_acc:.3f}, '
          f'leaves={tree.get_n_leaves()}')

# depth=2:  train~0.85 (underfit), few leaves
# depth=5:  train~0.92 (good fit), balanced leaves
# depth=20: train=1.00 (overfit), many tiny leaves

GridSearchCV for Tree Depth Hyperparameters

Use GridSearchCV to search over multiple regularisation parameters simultaneously. Searching over combinations of max_depth, min_samples_split, and min_samples_leaf finds the best-regularised tree in one step. This is more systematic than tuning parameters one at a time, as they can interact: a shallow tree may need larger min_samples_leaf than a deep tree. Always perform this search within a cross-validation loop to get an unbiased estimate of the optimal configuration's true performance.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'max_depth': [3, 5, 7, 10, None],
    'min_samples_split': [2, 10, 20],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best params:', grid.best_params_)
print('Best CV accuracy:', grid.best_score_.round(3))

Learning Curves: Diagnosing Over- and Underfitting

Learning curves plot model performance vs the number of training samples. For a well-tuned tree: both training and validation accuracy converge to a high value as training size increases. For an overfit deep tree: training accuracy stays high but validation accuracy remains low even with many samples. For an underfit shallow tree: both curves plateau at a mediocre accuracy regardless of data size. If adding more data improves validation accuracy, the model is underfitting. If the gap between train and val accuracy is large, the model is overfitting and needs more regularisation.

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

for depth in [3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    sizes, tr_sc, val_sc = learning_curve(
        tree, X, y, cv=5,
        train_sizes=np.linspace(0.1, 1.0, 10)
    )
    print(f'max_depth={depth}: final train={tr_sc[:,-1].mean():.3f}, '
          f'final val={val_sc[:,-1].mean():.3f}')

Best Practices for Tree Depth Control

A summary of depth-control best practices: (1) Always start with a shallow tree (depth 3-5) and add depth only if cross-validation improves. (2) Use min_samples_leaf in addition to max_depth — they control overfitting at different granularities. (3) For imbalanced datasets, use stratified CV when tuning. (4) If the dataset has many features, also consider max_features to introduce randomness. (5) Remember that if you plan to use an ensemble (Random Forest or Gradient Boosting), single-tree depth tuning is less critical — the ensemble will compensate.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Practical starting configuration
tree = DecisionTreeClassifier(
    max_depth=5,             # Start shallow
    min_samples_split=20,    # Need 20+ samples to split
    min_samples_leaf=10,     # Each leaf must have 10+ samples
    class_weight='balanced', # Handle class imbalance
    random_state=42
)

score = cross_val_score(tree, X, y, cv=10)
print(f'CV accuracy: {score.mean():.3f} (+/- {score.std():.3f})')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how unconstrained trees overfit by memorising training noise, the main regularisation parameters — max_depth, min_samples_split, min_samples_leaf, and max_leaf_nodes — and how to select them via GridSearchCV, and cost-complexity pruning as a post-training alternative. Next up we explore how to visualise and interpret decision trees to communicate model decisions.

Domande Frequenti

La lezione «Controllare la profondità dell'albero per prevenire l'overfitting» è gratuita?

Sì — il testo completo di «Controllare la profondità dell'albero per prevenire l'overfitting» è gratuito qui sul web. Per esercitarvi in modo interattivo (un editor di codice integrato e un tutor IA 24/7) e sbloccare il resto del corso Machine Learning Academy, passa a CoddyKit PRO. Il corso Machine Learning Academy include 4 lezioni in totale.

Cosa imparerò in «Controllare la profondità dell'albero per prevenire l'overfitting»?

Addestrerà alberi con diversi valori di max_depth, osserverà il compromesso tra overfitting e underfitting e sceglierà la profondità tramite il punteggio di validazione. Eserciti Machine Learning Academy con codice pratico che esegui direttamente nel browser, e un tutor IA 24/7 risponde alle tue domande mentre lavori sulla lezione.

Ho bisogno di esperienza per iniziare Machine Learning Academy?

Non è richiesta alcuna esperienza precedente. Machine Learning Academy su CoddyKit è strutturato per principianti e studenti avanzati, quindi puoi iniziare da qui o dall'inizio e procedere al tuo ritmo. Questa è la lezione 3 di 4.

Quanto tempo richiede la lezione «Controllare la profondità dell'albero per prevenire l'overfitting»?

La maggior parte delle lezioni CoddyKit richiede circa 5–10 minuti. Ogni lezione è breve e interattiva, quindi fai progressi costanti e riprendi esattamente da dove hai lasciato su web e app.

Posso scrivere ed eseguire codice in questa lezione Machine Learning Academy?

Sì. Ogni lezione Machine Learning Academy include un editor di codice integrato, quindi scrivi ed esegui codice reale direttamente nel tuo browser e ricevi feedback istantaneo dall'IA — nessuna configurazione locale necessaria.

Tutte le lezioni di questo corso

  1. Costruire un albero: suddivisioni, nodi e foglie
  2. Impurità di Gini e guadagno informativo
  3. Controllare la profondità dell'albero per prevenire l'overfitting
  4. Visualizzare e interpretare gli alberi decisionali
← Torna a Machine Learning Academy