0Pricing
Learn AI with Python · Lesson

Decision Trees: Theory and Implementation

Gini impurity, information gain, tree depth, overfitting — sklearn DecisionTreeClassifier.

Decision Trees: Theory and Implementation is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is a Decision Tree

A decision tree splits the data into branches based on feature values, asking yes/no questions until it reaches a prediction at a leaf node.

Each internal node tests one feature, each branch is an outcome, and each leaf assigns a class. Trees are easy to interpret because you can follow the path of decisions.

Gini Impurity

Gini impurity measures how mixed the classes are in a node. A pure node (all one class) has Gini 0.

The formula is Gini = 1 - sum(p_i^2) where p_i is the fraction of class i. The tree picks splits that reduce impurity the most.

import numpy as np

def gini(labels):
    classes, counts = np.unique(labels, return_counts=True)
    probs = counts / counts.sum()
    return 1 - np.sum(probs ** 2)

print(gini([0, 0, 1, 1]))   # 0.5 (max mix)
print(gini([0, 0, 0, 0]))   # 0.0 (pure)

Information Gain and Entropy

An alternative split criterion is information gain, based on entropy. Entropy is -sum(p_i * log2(p_i)).

Information gain = entropy(parent) - weighted entropy(children). Both Gini and entropy usually produce similar trees; Gini is slightly faster to compute.

import numpy as np

def entropy(labels):
    _, counts = np.unique(labels, return_counts=True)
    p = counts / counts.sum()
    return -np.sum(p * np.log2(p))

print(entropy([0, 0, 1, 1]))   # 1.0
print(entropy([0, 0, 0, 1]))   # ~0.81

Training a DecisionTreeClassifier

Scikit-learn provides DecisionTreeClassifier. You choose the split criterion with the criterion parameter (gini or entropy).

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)

clf = DecisionTreeClassifier(criterion="gini", random_state=0)
clf.fit(Xtr, ytr)
print("Accuracy:", clf.score(Xte, yte))

Overfitting and max_depth

An unconstrained tree grows until every leaf is pure, memorizing noise. This overfits.

The max_depth parameter limits how deep the tree can grow, forcing it to generalize. Smaller depth = simpler model = less overfitting.

from sklearn.tree import DecisionTreeClassifier

shallow = DecisionTreeClassifier(max_depth=3, random_state=0)
deep = DecisionTreeClassifier(max_depth=None, random_state=0)
# shallow generalizes better on unseen data;
# deep often overfits the training set

Other Pre-Pruning Parameters

Besides max_depth, you can control growth with:

  • min_samples_split minimum samples to split a node
  • min_samples_leaf minimum samples in a leaf
  • max_leaf_nodes cap on total leaves

These all reduce variance and combat overfitting.

from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier(
    max_depth=5,
    min_samples_split=10,
    min_samples_leaf=5,
    random_state=0,
)

Visualizing with plot_tree

plot_tree draws the full tree so you can read every split, the Gini value, and the class distribution at each node.

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
clf = DecisionTreeClassifier(max_depth=3).fit(X, y)

plt.figure(figsize=(12, 6))
plot_tree(clf, filled=True, feature_names=load_iris().feature_names)
plt.show()

Feature Importances

After fitting, feature_importances_ tells you how much each feature reduced impurity across all splits. Values sum to 1.0.

This is a fast way to rank which inputs matter most for the model.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

data = load_iris()
clf = DecisionTreeClassifier(max_depth=3).fit(data.data, data.target)

for name, imp in zip(data.feature_names, clf.feature_importances_):
    print(f"{name}: {imp:.3f}")

Cost-Complexity Pruning (ccp_alpha)

Post-pruning grows a full tree then trims weak branches. The ccp_alpha parameter controls how aggressively to prune: higher alpha removes more nodes.

Use cost_complexity_pruning_path to find candidate alphas.

from sklearn.tree import DecisionTreeClassifier

base = DecisionTreeClassifier(random_state=0)
path = base.cost_complexity_pruning_path(Xtr, ytr)
alphas = path.ccp_alphas

pruned = DecisionTreeClassifier(ccp_alpha=0.01, random_state=0)
pruned.fit(Xtr, ytr)

Choosing the Best Alpha

To pick ccp_alpha, train one tree per candidate alpha and compare validation accuracy. The best alpha balances accuracy and simplicity.

from sklearn.tree import DecisionTreeClassifier

scores = []
for a in alphas:
    t = DecisionTreeClassifier(ccp_alpha=a, random_state=0)
    t.fit(Xtr, ytr)
    scores.append((a, t.score(Xte, yte)))

best = max(scores, key=lambda s: s[1])
print("Best alpha:", best[0])

Strengths and Weaknesses

Pros: interpretable, no scaling needed, handles non-linear boundaries, mixed data types.

Cons: high variance (small data changes flip the tree), prone to overfitting, axis-aligned splits only. These weaknesses motivate ensembles like random forests.

Quick Check

Test your understanding of decision tree concepts.

Recap

Recap: Decision trees split data using Gini impurity or information gain. Control overfitting with pre-pruning (max_depth, min_samples_leaf) or post-pruning (ccp_alpha). Inspect models with plot_tree and feature_importances_. Their high variance motivates ensemble methods.

Frequently asked questions

Is the “Decision Trees: Theory and Implementation” lesson free?

Yes — the full text of “Decision Trees: Theory and Implementation” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Decision Trees: Theory and Implementation”?

Gini impurity, information gain, tree depth, overfitting — sklearn DecisionTreeClassifier. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Decision Trees: Theory and Implementation” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Decision Trees: Theory and Implementation
  2. Random Forests and Bagging
  3. Gradient Boosting: GBM and XGBoost
  4. LightGBM and CatBoost
← Back to Learn AI with Python