Machine Learning Academy · レッスン

木の深さを制御して過学習を防ぐ

max_depthを変えた木を訓練し、過学習と未学習のトレードオフを観察して、検証スコアで深さを選びます。

「木の深さを制御して過学習を防ぐ」はCoddyKit上の無料Machine Learning Academyレッスンです。これはレッスン3/4です。下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。これはMachine Learning Academy学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Machine Learning Academyコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

How Depth Leads to Overfitting

An unconstrained decision tree will grow until every training sample has its own leaf — achieving 100% training accuracy by memorising all data points, including noise. This is the extreme case of overfitting: the tree learns the idiosyncrasies of training data rather than general patterns. On any new data, such a tree performs poorly because its rules are too specific. Controlling tree depth is the primary regularisation mechanism for decision trees, analogous to choosing alpha in regularised linear models or choosing k in KNN.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Unlimited depth: memorises training data
tree = DecisionTreeClassifier(random_state=42)  # no max_depth
tree.fit(X_tr, y_tr)
print('Unlimited depth tree:')
print('  Tree depth:', tree.get_depth())
print('  Train accuracy:', tree.score(X_tr, y_tr).round(3))  # 1.000
print('  Test  accuracy:', tree.score(X_te, y_te).round(3))  # < 1.000

The max_depth Parameter

max_depth limits how many levels the tree can grow. With max_depth=1, the tree makes exactly one decision (a 'stump'). With max_depth=3, the tree can make up to three sequential questions. Shallower trees generalise better but may underfit; deeper trees fit training data better but risk overfitting. The right max_depth is a hyperparameter found through cross-validation. A useful heuristic: start around max_depth=3-5 and tune from there using a validation curve.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

for depth in [1, 2, 3, 5, 10, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_tr, y_tr)
    print(f'max_depth={str(depth):4}: train={tree.score(X_tr,y_tr):.3f}, '
          f'test={tree.score(X_te,y_te):.3f}')

Validation Curve for max_depth

Plotting training and validation accuracy vs max_depth reveals the classic bias-variance pattern. At depth 1, both training and validation accuracy are low (underfitting — high bias). As depth increases, training accuracy rises quickly to 100% while validation accuracy peaks and then decreases (overfitting — high variance). The optimal depth is where the validation curve peaks — before the train-test gap widens. Use cross-validation rather than a single validation split for a more reliable estimate of the peak location.

from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)

train_sc, val_sc = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y, param_name='max_depth',
    param_range=range(1, 16), cv=10
)

plt.plot(range(1,16), train_sc.mean(axis=1), label='Train')
plt.plot(range(1,16), val_sc.mean(axis=1), label='Validation')
plt.xlabel('max_depth'); plt.ylabel('Accuracy')
plt.title('Bias-Variance via max_depth')
plt.legend(); plt.show()

best_depth = np.argmax(val_sc.mean(axis=1)) + 1
print('Best depth:', best_depth)

min_samples_split: Minimum Samples to Split a Node

min_samples_split prevents splitting a node if it contains fewer than a specified number of samples. The default is 2 (any node with 2 or more samples can be split). Increasing this value forces the tree to wait for more evidence before making a decision, which prevents highly specific splits on tiny groups that are likely noise. Setting min_samples_split=20 means no node with fewer than 20 samples will be split further. This is especially useful for datasets with many samples and many rare subgroups.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_split in [2, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_split=min_split,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_split={min_split:3}: depth={tree.get_depth()}, CV acc={score:.3f}')

min_samples_leaf: Minimum Samples in a Leaf

min_samples_leaf requires that after any split, each resulting child node must have at least this many samples. If a candidate split would create a child with too few samples, that split is rejected. This is a stronger constraint than min_samples_split because it guarantees a minimum number of samples in every leaf. Larger min_samples_leaf produces smaller, less deep trees with smoother decision boundaries. For probability estimation, this parameter is critical: leaf nodes with very few samples produce unreliable probability estimates.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_leaf in [1, 5, 10, 20, 50]:
    tree = DecisionTreeClassifier(
        min_samples_leaf=min_leaf,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'min_samples_leaf={min_leaf:3}: leaves={tree.get_n_leaves():4}, CV acc={score:.3f}')

max_leaf_nodes: Limiting Total Leaves

Instead of controlling depth, you can directly limit the maximum number of leaf nodes. With max_leaf_nodes=10, the tree grows until it has exactly 10 leaves, choosing the split that provides the greatest impurity reduction at each step (best-first growth, not depth-first). This produces more balanced trees than depth-limited growth, as splits are allocated where they are most informative rather than being forced to equal depth. Best-first growth can sometimes give better performance than depth-limited growth on the same number of leaves.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for n_leaves in [2, 4, 8, 16, 32, 64]:
    tree = DecisionTreeClassifier(
        max_leaf_nodes=n_leaves,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=10).mean()
    tree.fit(X, y)
    print(f'max_leaf_nodes={n_leaves:3}: depth={tree.get_depth()}, '
          f'actual leaves={tree.get_n_leaves()}, CV acc={score:.3f}')

Cost-Complexity Pruning with ccp_alpha

Post-pruning reduces a fully-grown tree after training by removing splits that contribute little predictive value. Scikit-learn implements cost-complexity pruning via the ccp_alpha parameter. Larger ccp_alpha prunes more aggressively, resulting in smaller trees. You can find the optimal alpha by computing the pruning path with cost_complexity_pruning_path(), which returns the alpha values at which each subtree becomes optimal. Cross-validate over these alphas to find the one that maximises validation accuracy.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# Get pruning path
tree = DecisionTreeClassifier(random_state=42)
path = tree.cost_complexity_pruning_path(X_tr, y_tr)
ccp_alphas = path.ccp_alphas

# Cross-validate each alpha
best_alpha, best_score = 0, 0
for alpha in ccp_alphas[::5]:  # sample every 5th
    t = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42)
    score = cross_val_score(t, X_tr, y_tr, cv=5).mean()
    if score > best_score:
        best_score, best_alpha = score, alpha

print(f'Best ccp_alpha: {best_alpha:.5f}, CV acc: {best_score:.3f}')

Visualising Overfitting vs Good Fit

Plotting a shallow tree vs a deep tree side-by-side on a 2D dataset makes the overfitting effect visually obvious. A shallow tree (depth=2) draws a few rectangular decision regions that capture the main structure of the data. A deep tree (depth=20) creates hundreds of tiny rectangular regions that perfectly trace the training points, including noise. Good generalisation comes from finding the depth where test accuracy is highest — this is always below the fully-grown tree depth, and finding it is the goal of every depth-tuning experiment.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
import numpy as np

X, y = make_moons(n_samples=300, noise=0.3, random_state=42)

for depth in [2, 5, 20]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X, y)
    train_acc = tree.score(X, y)
    print(f'depth={depth:2}: train={train_acc:.3f}, '
          f'leaves={tree.get_n_leaves()}')

# depth=2:  train~0.85 (underfit), few leaves
# depth=5:  train~0.92 (good fit), balanced leaves
# depth=20: train=1.00 (overfit), many tiny leaves

GridSearchCV for Tree Depth Hyperparameters

Use GridSearchCV to search over multiple regularisation parameters simultaneously. Searching over combinations of max_depth, min_samples_split, and min_samples_leaf finds the best-regularised tree in one step. This is more systematic than tuning parameters one at a time, as they can interact: a shallow tree may need larger min_samples_leaf than a deep tree. Always perform this search within a cross-validation loop to get an unbiased estimate of the optimal configuration's true performance.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'max_depth': [3, 5, 7, 10, None],
    'min_samples_split': [2, 10, 20],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best params:', grid.best_params_)
print('Best CV accuracy:', grid.best_score_.round(3))

Learning Curves: Diagnosing Over- and Underfitting

Learning curves plot model performance vs the number of training samples. For a well-tuned tree: both training and validation accuracy converge to a high value as training size increases. For an overfit deep tree: training accuracy stays high but validation accuracy remains low even with many samples. For an underfit shallow tree: both curves plateau at a mediocre accuracy regardless of data size. If adding more data improves validation accuracy, the model is underfitting. If the gap between train and val accuracy is large, the model is overfitting and needs more regularisation.

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

for depth in [3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    sizes, tr_sc, val_sc = learning_curve(
        tree, X, y, cv=5,
        train_sizes=np.linspace(0.1, 1.0, 10)
    )
    print(f'max_depth={depth}: final train={tr_sc[:,-1].mean():.3f}, '
          f'final val={val_sc[:,-1].mean():.3f}')

Best Practices for Tree Depth Control

A summary of depth-control best practices: (1) Always start with a shallow tree (depth 3-5) and add depth only if cross-validation improves. (2) Use min_samples_leaf in addition to max_depth — they control overfitting at different granularities. (3) For imbalanced datasets, use stratified CV when tuning. (4) If the dataset has many features, also consider max_features to introduce randomness. (5) Remember that if you plan to use an ensemble (Random Forest or Gradient Boosting), single-tree depth tuning is less critical — the ensemble will compensate.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Practical starting configuration
tree = DecisionTreeClassifier(
    max_depth=5,             # Start shallow
    min_samples_split=20,    # Need 20+ samples to split
    min_samples_leaf=10,     # Each leaf must have 10+ samples
    class_weight='balanced', # Handle class imbalance
    random_state=42
)

score = cross_val_score(tree, X, y, cv=10)
print(f'CV accuracy: {score.mean():.3f} (+/- {score.std():.3f})')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how unconstrained trees overfit by memorising training noise, the main regularisation parameters — max_depth, min_samples_split, min_samples_leaf, and max_leaf_nodes — and how to select them via GridSearchCV, and cost-complexity pruning as a post-training alternative. Next up we explore how to visualise and interpret decision trees to communicate model decisions.

よくある質問

「木の深さを制御して過学習を防ぐ」レッスンは無料ですか？

はい。「木の深さを制御して過学習を防ぐ」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Machine Learning Academyコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Machine Learning Academyコースには全4レッスンが含まれています。

「木の深さを制御して過学習を防ぐ」で何を学びますか？

max_depthを変えた木を訓練し、過学習と未学習のトレードオフを観察して、検証スコアで深さを選びます。ブラウザで直接実行するハンズオンコードでMachine Learning Academyを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

Machine Learning Academyを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのMachine Learning Academyは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。これはレッスン3/4です。

「木の深さを制御して過学習を防ぐ」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このMachine Learning Academyレッスンでコードを書いて実行できますか？

はい。すべてのMachine Learning Academyレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

← Machine Learning Academyに戻る