Machine Learning Academy · 课时

控制树深度以防止过拟合

您将训练 max_depth 不同的树，观察过拟合与欠拟合之间的权衡，并通过验证分数选择深度

第 3 / 4 课13 个步骤

控制树深度以防止过拟合 是 CoddyKit 上的免费 Machine Learning Academy 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

How Depth Leads to Overfitting

An unconstrained decision tree will grow until every training sample has its own leaf — achieving 100% training accuracy by memorising all data points, including noise. This is the extreme case of overfitting: the tree learns the idiosyncrasies of training data rather than general patterns. On any new data, such a tree performs poorly because its rules are too specific. Controlling tree depth is the primary regularisation mechanism for decision trees, analogous to choosing alpha in regularised linear models or choosing k in KNN.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Unlimited depth: memorises training data
tree = DecisionTreeClassifier(random_state=42)  # no max_depth
tree.fit(X_tr, y_tr)
print('Unlimited depth tree:')
print('  Tree depth:', tree.get_depth())
print('  Train accuracy:', tree.score(X_tr, y_tr).round(3))  # 1.000
print('  Test  accuracy:', tree.score(X_te, y_te).round(3))  # < 1.000

The max_depth Parameter

max_depth limits how many levels the tree can grow. With max_depth=1, the tree makes exactly one decision (a 'stump'). With max_depth=3, the tree can make up to three sequential questions. Shallower trees generalise better but may underfit; deeper trees fit training data better but risk overfitting. The right max_depth is a hyperparameter found through cross-validation. A useful heuristic: start around max_depth=3-5 and tune from there using a validation curve.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

for depth in [1, 2, 3, 5, 10, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_tr, y_tr)
    print(f'max_depth={str(depth):4}: train={tree.score(X_tr,y_tr):.3f}, '
          f'test={tree.score(X_te,y_te):.3f}')

Validation Curve for max_depth

Plotting training and validation accuracy vs max_depth reveals the classic bias-variance pattern. At depth 1, both training and validation accuracy are low (underfitting — high bias). As depth increases, training accuracy rises quickly to 100% while validation accuracy peaks and then decreases (overfitting — high variance). The optimal depth is where the validation curve peaks — before the train-test gap widens. Use cross-validation rather than a single validation split for a more reliable estimate of the peak location.

from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)

train_sc, val_sc = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y, param_name='max_depth',
    param_range=range(1, 16), cv=10
)

plt.plot(range(1,16), train_sc.mean(axis=1), label='Train')
plt.plot(range(1,16), val_sc.mean(axis=1), label='Validation')
plt.xlabel('max_depth'); plt.ylabel('Accuracy')
plt.title('Bias-Variance via max_depth')
plt.legend(); plt.show()

best_depth = np.argmax(val_sc.mean(axis=1)) + 1
print('Best depth:', best_depth)

min_samples_split: Minimum Samples to Split a Node

min_samples_split prevents splitting a node if it contains fewer than a specified number of samples. The default is 2 (any node with 2 or more samples can be split). Increasing this value forces the tree to wait for more evidence before making a decision, which prevents highly specific splits on tiny groups that are likely noise. Setting min_samples_split=20 means no node with fewer than 20 samples will be split further. This is especially useful for datasets with many samples and many rare subgroups.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_split in [2, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_split=min_split,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_split={min_split:3}: depth={tree.get_depth()}, CV acc={score:.3f}')

min_samples_leaf: Minimum Samples in a Leaf

min_samples_leaf requires that after any split, each resulting child node must have at least this many samples. If a candidate split would create a child with too few samples, that split is rejected. This is a stronger constraint than min_samples_split because it guarantees a minimum number of samples in every leaf. Larger min_samples_leaf produces smaller, less deep trees with smoother decision boundaries. For probability estimation, this parameter is critical: leaf nodes with very few samples produce unreliable probability estimates.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_leaf in [1, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_leaf=min_leaf,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_leaf={min_leaf:3}: leaves={tree.get_n_leaves():4}, CV acc={score:.3f}')

max_leaf_nodes: Limiting Total Leaves

Instead of controlling depth, you can directly limit the maximum number of leaf nodes. With max_leaf_nodes=10, the tree grows until it has exactly 10 leaves, choosing the split that provides the greatest impurity reduction at each step (best-first growth, not depth-first). This produces more balanced trees than depth-limited growth, as splits are allocated where they are most informative rather than being forced to equal depth. Best-first growth can sometimes give better performance than depth-limited growth on the same number of leaves.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for n_leaves in [2, 4, 8, 16, 32, 64]:
    tree = DecisionTreeClassifier(
        max_leaf_nodes=n_leaves,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'max_leaf_nodes={n_leaves:3}: depth={tree.get_depth()}, '
          f'actual leaves={tree.get_n_leaves()}, CV acc={score:.3f}')

Cost-Complexity Pruning with ccp_alpha

Post-pruning reduces a fully-grown tree after training by removing splits that contribute little predictive value. Scikit-learn implements cost-complexity pruning via the ccp_alpha parameter. Larger ccp_alpha prunes more aggressively, resulting in smaller trees. You can find the optimal alpha by computing the pruning path with cost_complexity_pruning_path(), which returns the alpha values at which each subtree becomes optimal. Cross-validate over these alphas to find the one that maximises validation accuracy.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Get pruning path
tree = DecisionTreeClassifier(random_state=42)
path = tree.cost_complexity_pruning_path(X_tr, y_tr)
ccp_alphas = path.ccp_alphas

# Cross-validate each alpha
best_alpha, best_score = 0, 0
for alpha in ccp_alphas[::5]:  # sample every 5th
    t = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42)
    score = cross_val_score(t, X_tr, y_tr, cv=5).mean()
    if score > best_score:
        best_score, best_alpha = score, alpha

print(f'Best ccp_alpha: {best_alpha:.5f}, CV acc: {best_score:.3f}')

Visualising Overfitting vs Good Fit

Plotting a shallow tree vs a deep tree side-by-side on a 2D dataset makes the overfitting effect visually obvious. A shallow tree (depth=2) draws a few rectangular decision regions that capture the main structure of the data. A deep tree (depth=20) creates hundreds of tiny rectangular regions that perfectly trace the training points, including noise. Good generalisation comes from finding the depth where test accuracy is highest — this is always below the fully-grown tree depth, and finding it is the goal of every depth-tuning experiment.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
import numpy as np

X, y = make_moons(n_samples=300, noise=0.3, random_state=42)

for depth in [2, 5, 20]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X, y)
    train_acc = tree.score(X, y)
    print(f'depth={depth:2}: train={train_acc:.3f}, '
          f'leaves={tree.get_n_leaves()}')

# depth=2:  train~0.85 (underfit), few leaves
# depth=5:  train~0.92 (good fit), balanced leaves
# depth=20: train=1.00 (overfit), many tiny leaves

GridSearchCV for Tree Depth Hyperparameters

Use GridSearchCV to search over multiple regularisation parameters simultaneously. Searching over combinations of max_depth, min_samples_split, and min_samples_leaf finds the best-regularised tree in one step. This is more systematic than tuning parameters one at a time, as they can interact: a shallow tree may need larger min_samples_leaf than a deep tree. Always perform this search within a cross-validation loop to get an unbiased estimate of the optimal configuration's true performance.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'max_depth': [3, 5, 7, 10, None],
    'min_samples_split': [2, 10, 20],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best params:', grid.best_params_)
print('Best CV accuracy:', grid.best_score_.round(3))

Learning Curves: Diagnosing Over- and Underfitting

Learning curves plot model performance vs the number of training samples. For a well-tuned tree: both training and validation accuracy converge to a high value as training size increases. For an overfit deep tree: training accuracy stays high but validation accuracy remains low even with many samples. For an underfit shallow tree: both curves plateau at a mediocre accuracy regardless of data size. If adding more data improves validation accuracy, the model is underfitting. If the gap between train and val accuracy is large, the model is overfitting and needs more regularisation.

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

for depth in [3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    sizes, tr_sc, val_sc = learning_curve(
        tree, X, y, cv=5,
        train_sizes=np.linspace(0.1, 1.0, 10)
    )
    print(f'max_depth={depth}: final train={tr_sc[:,-1].mean():.3f}, '
          f'final val={val_sc[:,-1].mean():.3f}')

Best Practices for Tree Depth Control

A summary of depth-control best practices: (1) Always start with a shallow tree (depth 3-5) and add depth only if cross-validation improves. (2) Use min_samples_leaf in addition to max_depth — they control overfitting at different granularities. (3) For imbalanced datasets, use stratified CV when tuning. (4) If the dataset has many features, also consider max_features to introduce randomness. (5) Remember that if you plan to use an ensemble (Random Forest or Gradient Boosting), single-tree depth tuning is less critical — the ensemble will compensate.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Practical starting configuration
tree = DecisionTreeClassifier(
    max_depth=5,             # Start shallow
    min_samples_split=20,    # Need 20+ samples to split
    min_samples_leaf=10,     # Each leaf must have 10+ samples
    class_weight='balanced', # Handle class imbalance
    random_state=42
)

score = cross_val_score(tree, X, y, cv=10)
print(f'CV accuracy: {score.mean():.3f} (+/- {score.std():.3f})')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

在交互式课程中回答

Lesson Recap

In this lesson you learned: how unconstrained trees overfit by memorising training noise, the main regularisation parameters — max_depth, min_samples_split, min_samples_leaf, and max_leaf_nodes — and how to select them via GridSearchCV, and cost-complexity pruning as a post-training alternative. Next up we explore how to visualise and interpret decision trees to communicate model decisions.

免费开始

用 AI 导师学习 Python — 免费

在浏览器中编写并运行真实代码，获得全天候 AI 导师的即时帮助，并在网页或应用中继续学习。

课程: 30
课程: 120

免费开始学习查看完整课程

常见问题解答

「控制树深度以防止过拟合」课时是免费的吗？

是的 — 「控制树深度以防止过拟合」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「控制树深度以防止过拟合」这节课中我会学到什么？

您将训练 max_depth 不同的树，观察过拟合与欠拟合之间的权衡，并通过验证分数选择深度你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「控制树深度以防止过拟合」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Machine Learning Academy