Machine Learning Academy · Урок

Управление глубиной дерева для предотвращения переобучения

Вы обучите деревья с разными значениями max_depth, изучите компромисс между переобучением и недообучением и выберете глубину по оценке на проверочной выборке

«Управление глубиной дерева для предотвращения переобучения» — бесплатный урок Machine Learning Academy на CoddyKit. Это урок 3 из 4. Ты можешь прочитать весь урок бесплатно ниже — а потом практиковать его прямо в браузере с встроенным редактором кода и ИИ-репетитором 24/7. Это часть пути обучения Machine Learning Academy, и твой прогресс синхронизируется между веб-версией и приложением CoddyKit. Курс Machine Learning Academy содержит 4 уроков всего.

Части этого урока еще не переведены и отображаются на английском.

How Depth Leads to Overfitting

An unconstrained decision tree will grow until every training sample has its own leaf — achieving 100% training accuracy by memorising all data points, including noise. This is the extreme case of overfitting: the tree learns the idiosyncrasies of training data rather than general patterns. On any new data, such a tree performs poorly because its rules are too specific. Controlling tree depth is the primary regularisation mechanism for decision trees, analogous to choosing alpha in regularised linear models or choosing k in KNN.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Unlimited depth: memorises training data
tree = DecisionTreeClassifier(random_state=42)  # no max_depth
tree.fit(X_tr, y_tr)
print('Unlimited depth tree:')
print('  Tree depth:', tree.get_depth())
print('  Train accuracy:', tree.score(X_tr, y_tr).round(3))  # 1.000
print('  Test  accuracy:', tree.score(X_te, y_te).round(3))  # < 1.000

The max_depth Parameter

max_depth limits how many levels the tree can grow. With max_depth=1, the tree makes exactly one decision (a 'stump'). With max_depth=3, the tree can make up to three sequential questions. Shallower trees generalise better but may underfit; deeper trees fit training data better but risk overfitting. The right max_depth is a hyperparameter found through cross-validation. A useful heuristic: start around max_depth=3-5 and tune from there using a validation curve.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

for depth in [1, 2, 3, 5, 10, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_tr, y_tr)
    print(f'max_depth={str(depth):4}: train={tree.score(X_tr,y_tr):.3f}, '
          f'test={tree.score(X_te,y_te):.3f}')

Validation Curve for max_depth

Plotting training and validation accuracy vs max_depth reveals the classic bias-variance pattern. At depth 1, both training and validation accuracy are low (underfitting — high bias). As depth increases, training accuracy rises quickly to 100% while validation accuracy peaks and then decreases (overfitting — high variance). The optimal depth is where the validation curve peaks — before the train-test gap widens. Use cross-validation rather than a single validation split for a more reliable estimate of the peak location.

from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)

train_sc, val_sc = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y, param_name='max_depth',
    param_range=range(1, 16), cv=10
)

plt.plot(range(1,16), train_sc.mean(axis=1), label='Train')
plt.plot(range(1,16), val_sc.mean(axis=1), label='Validation')
plt.xlabel('max_depth'); plt.ylabel('Accuracy')
plt.title('Bias-Variance via max_depth')
plt.legend(); plt.show()

best_depth = np.argmax(val_sc.mean(axis=1)) + 1
print('Best depth:', best_depth)

min_samples_split: Minimum Samples to Split a Node

min_samples_split prevents splitting a node if it contains fewer than a specified number of samples. The default is 2 (any node with 2 or more samples can be split). Increasing this value forces the tree to wait for more evidence before making a decision, which prevents highly specific splits on tiny groups that are likely noise. Setting min_samples_split=20 means no node with fewer than 20 samples will be split further. This is especially useful for datasets with many samples and many rare subgroups.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_split in [2, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_split=min_split,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_split={min_split:3}: depth={tree.get_depth()}, CV acc={score:.3f}')

min_samples_leaf: Minimum Samples in a Leaf

min_samples_leaf requires that after any split, each resulting child node must have at least this many samples. If a candidate split would create a child with too few samples, that split is rejected. This is a stronger constraint than min_samples_split because it guarantees a minimum number of samples in every leaf. Larger min_samples_leaf produces smaller, less deep trees with smoother decision boundaries. For probability estimation, this parameter is critical: leaf nodes with very few samples produce unreliable probability estimates.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_leaf in [1, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_leaf=min_leaf,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_leaf={min_leaf:3}: leaves={tree.get_n_leaves():4}, CV acc={score:.3f}')

max_leaf_nodes: Limiting Total Leaves

Instead of controlling depth, you can directly limit the maximum number of leaf nodes. With max_leaf_nodes=10, the tree grows until it has exactly 10 leaves, choosing the split that provides the greatest impurity reduction at each step (best-first growth, not depth-first). This produces more balanced trees than depth-limited growth, as splits are allocated where they are most informative rather than being forced to equal depth. Best-first growth can sometimes give better performance than depth-limited growth on the same number of leaves.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for n_leaves in [2, 4, 8, 16, 32, 64]:
    tree = DecisionTreeClassifier(
        max_leaf_nodes=n_leaves,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'max_leaf_nodes={n_leaves:3}: depth={tree.get_depth()}, '
          f'actual leaves={tree.get_n_leaves()}, CV acc={score:.3f}')

Cost-Complexity Pruning with ccp_alpha

Post-pruning reduces a fully-grown tree after training by removing splits that contribute little predictive value. Scikit-learn implements cost-complexity pruning via the ccp_alpha parameter. Larger ccp_alpha prunes more aggressively, resulting in smaller trees. You can find the optimal alpha by computing the pruning path with cost_complexity_pruning_path(), which returns the alpha values at which each subtree becomes optimal. Cross-validate over these alphas to find the one that maximises validation accuracy.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Get pruning path
tree = DecisionTreeClassifier(random_state=42)
path = tree.cost_complexity_pruning_path(X_tr, y_tr)
ccp_alphas = path.ccp_alphas

# Cross-validate each alpha
best_alpha, best_score = 0, 0
for alpha in ccp_alphas[::5]:  # sample every 5th
    t = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42)
    score = cross_val_score(t, X_tr, y_tr, cv=5).mean()
    if score > best_score:
        best_score, best_alpha = score, alpha

print(f'Best ccp_alpha: {best_alpha:.5f}, CV acc: {best_score:.3f}')

Visualising Overfitting vs Good Fit

Plotting a shallow tree vs a deep tree side-by-side on a 2D dataset makes the overfitting effect visually obvious. A shallow tree (depth=2) draws a few rectangular decision regions that capture the main structure of the data. A deep tree (depth=20) creates hundreds of tiny rectangular regions that perfectly trace the training points, including noise. Good generalisation comes from finding the depth where test accuracy is highest — this is always below the fully-grown tree depth, and finding it is the goal of every depth-tuning experiment.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
import numpy as np

X, y = make_moons(n_samples=300, noise=0.3, random_state=42)

for depth in [2, 5, 20]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X, y)
    train_acc = tree.score(X, y)
    print(f'depth={depth:2}: train={train_acc:.3f}, '
          f'leaves={tree.get_n_leaves()}')

# depth=2:  train~0.85 (underfit), few leaves
# depth=5:  train~0.92 (good fit), balanced leaves
# depth=20: train=1.00 (overfit), many tiny leaves

GridSearchCV for Tree Depth Hyperparameters

Use GridSearchCV to search over multiple regularisation parameters simultaneously. Searching over combinations of max_depth, min_samples_split, and min_samples_leaf finds the best-regularised tree in one step. This is more systematic than tuning parameters one at a time, as they can interact: a shallow tree may need larger min_samples_leaf than a deep tree. Always perform this search within a cross-validation loop to get an unbiased estimate of the optimal configuration's true performance.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'max_depth': [3, 5, 7, 10, None],
    'min_samples_split': [2, 10, 20],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best params:', grid.best_params_)
print('Best CV accuracy:', grid.best_score_.round(3))

Learning Curves: Diagnosing Over- and Underfitting

Learning curves plot model performance vs the number of training samples. For a well-tuned tree: both training and validation accuracy converge to a high value as training size increases. For an overfit deep tree: training accuracy stays high but validation accuracy remains low even with many samples. For an underfit shallow tree: both curves plateau at a mediocre accuracy regardless of data size. If adding more data improves validation accuracy, the model is underfitting. If the gap between train and val accuracy is large, the model is overfitting and needs more regularisation.

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

for depth in [3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    sizes, tr_sc, val_sc = learning_curve(
        tree, X, y, cv=5,
        train_sizes=np.linspace(0.1, 1.0, 10)
    )
    print(f'max_depth={depth}: final train={tr_sc[:,-1].mean():.3f}, '
          f'final val={val_sc[:,-1].mean():.3f}')

Best Practices for Tree Depth Control

A summary of depth-control best practices: (1) Always start with a shallow tree (depth 3-5) and add depth only if cross-validation improves. (2) Use min_samples_leaf in addition to max_depth — they control overfitting at different granularities. (3) For imbalanced datasets, use stratified CV when tuning. (4) If the dataset has many features, also consider max_features to introduce randomness. (5) Remember that if you plan to use an ensemble (Random Forest or Gradient Boosting), single-tree depth tuning is less critical — the ensemble will compensate.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Practical starting configuration
tree = DecisionTreeClassifier(
    max_depth=5,             # Start shallow
    min_samples_split=20,    # Need 20+ samples to split
    min_samples_leaf=10,     # Each leaf must have 10+ samples
    class_weight='balanced', # Handle class imbalance
    random_state=42
)

score = cross_val_score(tree, X, y, cv=10)
print(f'CV accuracy: {score.mean():.3f} (+/- {score.std():.3f})')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how unconstrained trees overfit by memorising training noise, the main regularisation parameters — max_depth, min_samples_split, min_samples_leaf, and max_leaf_nodes — and how to select them via GridSearchCV, and cost-complexity pruning as a post-training alternative. Next up we explore how to visualise and interpret decision trees to communicate model decisions.

Часто задаваемые вопросы

Урок «Управление глубиной дерева для предотвращения переобучения» бесплатный?

Да — полный текст урока «Управление глубиной дерева для предотвращения переобучения» бесплатно доступен здесь в веб-версии. Чтобы практиковать его интерактивно (встроенный редактор кода и ИИ-репетитор 24/7) и разблокировать остальной курс Machine Learning Academy, подпишись на CoddyKit PRO. Курс Machine Learning Academy содержит 4 уроков всего.

Чему я научусь в уроке «Управление глубиной дерева для предотвращения переобучения»?

Вы обучите деревья с разными значениями max_depth, изучите компромисс между переобучением и недообучением и выберете глубину по оценке на проверочной выборке Ты практикуешь Machine Learning Academy с помощью реального кода, который запускаешь прямо в браузере, и ИИ-репетитор 24/7 отвечает на твои вопросы во время урока.

Нужен ли мне опыт, чтобы начать Machine Learning Academy?

Предыдущий опыт не требуется. Machine Learning Academy на CoddyKit структурирован для всех уровней — от новичков до продвинутых, поэтому ты можешь начать отсюда или с самого начала и учиться в своем темпе. Это урок 3 из 4.

Сколько времени занимает урок «Управление глубиной дерева для предотвращения переобучения»?

Большинство уроков CoddyKit занимают около 5–10 минут. Каждый из них компактный и интерактивный, поэтому ты постоянно делаешь прогресс и продолжаешь с того же места в веб-версии и приложении.

Можно ли писать и запускать код в этом уроке Machine Learning Academy?

Да. Каждый урок Machine Learning Academy включает встроенный редактор кода, поэтому ты пишешь и запускаешь реальный код прямо в браузере и получаешь моментальную обратную связь от AI — локальная установка не требуется.

Все уроки этого курса

Построение дерева: разбиения, узлы и листья
Неоднородность Джини и прирост информации
Управление глубиной дерева для предотвращения переобучения
Визуализация и интерпретация деревьев решений

← Назад к Machine Learning Academy