0Pricing
Machine Learning Academy · Lesson

Gini Impurity and Information Gain

Learners will calculate Gini impurity and entropy for sample splits, understand why the tree picks the split that maximises information gain.

Gini Impurity and Information Gain is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Splitting Problem: Which Feature to Ask About?

When building a decision tree, at each node we must choose which feature and which threshold produces the most useful split. The goal is to create child nodes where samples are as pure as possible — ideally each child contains only one class. We need a mathematical measure of impurity that tells us how mixed the classes are in a node. Lower impurity is better: a node with all samples belonging to one class has zero impurity (perfect purity). Two widely-used impurity measures are Gini impurity and entropy.

# Impurity measures how mixed the classes are in a node
# Perfect purity: all samples belong to one class -> impurity = 0
# Maximum impurity: classes are equally distributed

import numpy as np

# Node A: all class 0 -> pure
node_a = [0, 0, 0, 0]  # impurity = 0

# Node B: 50/50 mix -> maximally impure
node_b = [0, 0, 1, 1]  # impurity = maximum

# Node C: mostly one class
node_c = [0, 0, 0, 1]  # impurity = low

for name, node in [('A', node_a), ('B', node_b), ('C', node_c)]:
    print(f'Node {name}: classes = {node}')

Gini Impurity: The Default Criterion

Gini impurity measures the probability that a randomly chosen sample from a node would be incorrectly labelled if labelled randomly according to the class distribution in that node. The formula is: Gini = 1 - sum(p_i^2) where p_i is the proportion of class i. Gini ranges from 0 (pure) to 0.5 (two-class equal split). For K classes, the maximum is 1 - 1/K. Gini impurity is the default criterion in scikit-learn's DecisionTreeClassifier because it is computationally efficient (no logarithms).

import numpy as np

def gini_impurity(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    return 1 - np.sum(probabilities ** 2)

# Pure node
print('Pure [0,0,0,0]:', gini_impurity([0,0,0,0]))       # 0.0

# 50/50 split
print('50/50 [0,0,1,1]:', gini_impurity([0,0,1,1]))      # 0.5

# 75/25 split
print('75/25 [0,0,0,1]:', gini_impurity([0,0,0,1]))      # 0.375

# Three classes equal
print('3-class equal:', gini_impurity([0,1,2,0,1,2]))    # ~0.667

Entropy and Information Theory

Entropy is borrowed from information theory: it measures the uncertainty or information content of a distribution. Formula: H = -sum(p_i * log2(p_i)). A pure node has entropy 0 (no uncertainty). A 50/50 split has entropy 1 (one bit of uncertainty — you need one question to determine the class). Entropy and Gini produce very similar trees in practice. Entropy is slightly slower to compute (requires log) but may give better splits when class distributions are skewed. Use criterion='entropy' in scikit-learn to switch.

import numpy as np

def entropy(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    # Avoid log(0) by filtering zero probabilities
    probs = probabilities[probabilities > 0]
    return -np.sum(probs * np.log2(probs))

print('Pure [0,0,0,0]:', entropy([0,0,0,0]))         # 0.0
print('50/50 [0,0,1,1]:', entropy([0,0,1,1]))        # 1.0 (1 bit)
print('75/25 [0,0,0,1]:', entropy([0,0,0,1]).round(3))  # 0.811
print('3-class equal:', entropy([0,1,2,0,1,2]).round(3)) # 1.585

Information Gain: The Split Quality Metric

Information gain measures how much a split reduces impurity. It is computed as the impurity of the parent node minus the weighted average impurity of the child nodes: IG = impurity(parent) - (N_left/N * impurity(left) + N_right/N * impurity(right)). The best split maximises information gain: it produces children that are as pure as possible, weighted by their size (so larger children count more). The tree builder evaluates information gain for every feature and every threshold, then picks the combination with the highest gain.

import numpy as np

def gini_impurity(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

def information_gain(y_parent, y_left, y_right):
    n = len(y_parent)
    n_l, n_r = len(y_left), len(y_right)
    parent_impurity = gini_impurity(y_parent)
    weighted_child  = (n_l/n)*gini_impurity(y_left) + (n_r/n)*gini_impurity(y_right)
    return parent_impurity - weighted_child

y_parent = [0,0,0,1,1,1]  # 50/50 parent
y_left   = [0,0,0]         # pure left
y_right  = [1,1,1]         # pure right

print('IG:', information_gain(y_parent, y_left, y_right))  # 0.5 (perfect split)

Evaluating Multiple Splits

To find the best split, the algorithm evaluates all candidate feature-threshold combinations and picks the one with the highest information gain. For a dataset with N samples and d features, the tree evaluates up to N-1 thresholds per feature (midpoints between consecutive unique values), giving O(N * d) splits to evaluate per node. Here is a simplified example showing how different thresholds produce different information gains on the same feature.

import numpy as np

X_feature = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 0, 1, 1, 1])

best_threshold, best_ig = None, -1

for threshold in [1.5, 2.5, 3.5, 4.5, 5.5]:
    left_mask  = X_feature <= threshold
    right_mask = ~left_mask
    y_l, y_r = y[left_mask], y[right_mask]
    
    _, cnt_p = np.unique(y, return_counts=True)
    _, cnt_l = np.unique(y_l, return_counts=True) if len(y_l) else (None, [1])
    _, cnt_r = np.unique(y_r, return_counts=True) if len(y_r) else (None, [1])
    
    ig = information_gain(y, y_l, y_r)
    print(f'Threshold {threshold}: IG = {ig:.3f}')
    if ig > best_ig:
        best_ig, best_threshold = ig, threshold

print('Best threshold:', best_threshold, 'with IG:', best_ig)

Gini vs Entropy: Practical Difference

Gini impurity and entropy produce nearly identical trees most of the time. The key differences are subtle: entropy tends to produce more balanced trees (it penalises imbalanced splits more heavily due to the logarithm), while Gini tends to isolate the most frequent class in one branch. Computationally, Gini is faster because it avoids the logarithm computation. In practice, the choice between them is a hyperparameter to tune — try both with cross-validation and pick the one that performs better on your specific dataset.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for criterion in ['gini', 'entropy']:
    tree = DecisionTreeClassifier(criterion=criterion, max_depth=5, random_state=42)
    score = cross_val_score(tree, X, y, cv=10).mean()
    print(f'criterion={criterion}: CV accuracy = {score:.3f}')

# Usually within 0.5% of each other -- not the critical choice

Weighted Gini for Multi-Class Problems

Gini impurity extends naturally to multi-class problems with no modification: Gini = 1 - sum(p_i^2) works for any number of classes. Information gain computation is also unchanged — weighted impurity of children minus parent impurity. For a 3-class problem, a perfectly pure leaf (all class 2, for example) has Gini 0. A node with equal proportions of 3 classes has maximum Gini of 2/3. Decision trees are one of the few algorithms that handle multi-class problems natively without any modification — unlike logistic regression, which requires one-vs-rest or softmax extensions.

import numpy as np

def gini_multiclass(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

# 3-class examples
print('Pure [0,0,0]:', gini_multiclass([0,0,0]))               # 0.0
print('Equal [0,1,2]:', gini_multiclass([0,1,2]).round(3))    # 0.667
print('2 dominant [0,0,1,2]:', gini_multiclass([0,0,1,2]).round(3))  # 0.625

# Max Gini for K classes = 1 - 1/K
for K in [2, 3, 4, 5]:
    print(f'Max Gini for {K} classes: {1 - 1/K:.3f}')

Impurity Reduction Inside scikit-learn

Inside scikit-learn, at each node the tree stores the impurity before the split and the impurity of each child. The difference, weighted by sample count, is the impurity reduction (information gain). This value is summed per feature across all nodes and normalised to compute feature_importances_ — the total impurity reduction attributed to each feature. Features that appear near the root and handle many samples tend to have the highest importance because each split affects a large fraction of the data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
import numpy as np

X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=42)
tree.fit(X, y)

# Impurity at root and children
print('Root impurity (Gini):', tree.tree_.impurity[0].round(4))
print('Left child impurity:', tree.tree_.impurity[1].round(4))
print('Right child impurity:', tree.tree_.impurity[2].round(4))

# Feature importances = total weighted impurity reduction per feature
print('Feature importances:', tree.feature_importances_.round(3))

The Role of min_impurity_decrease

By default, the tree splits nodes as long as there is any reduction in impurity and the node has enough samples. The min_impurity_decrease parameter adds a minimum threshold: a split is only created if it reduces impurity by at least this amount. This prevents the tree from making trivially small splits that memorise noise. Setting min_impurity_decrease=0.01 means the tree only splits when the information gain exceeds 0.01. This is a useful regularisation alternative to controlling depth directly.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_ig in [0.0, 0.001, 0.005, 0.01, 0.05]:
    tree = DecisionTreeClassifier(
        min_impurity_decrease=min_ig,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=5).mean()
    depth = tree.fit(X, y).get_depth()
    print(f'min_impurity_decrease={min_ig}: depth={depth}, CV acc={score:.3f}')

Comparing Splits Across Features

A worked example showing how the tree picks the best feature and threshold. Given two features and one split threshold each, the tree computes information gain for all options and picks the winner. This illustrates why trees naturally perform implicit feature selection: features that never produce high information gain at any threshold will never be chosen as split features, effectively being ignored. This makes decision trees robust to irrelevant features, unlike KNN which is harmed by them.

import numpy as np

# Toy dataset: X[:,0]=income, X[:,1]=age; y=churn
X = np.array([[100, 25], [120, 30], [40, 22], [50, 28], [90, 35], [30, 40]])
y = np.array([0, 0, 1, 1, 0, 1])

# Try splitting on income at 75
left_y = y[X[:, 0] <= 75]   # [1,1,1]
right_y = y[X[:, 0] > 75]   # [0,0,0]
ig_income = information_gain(y, left_y, right_y)

# Try splitting on age at 30
left_y2 = y[X[:, 1] <= 30]  # [0,0,1,1]
right_y2 = y[X[:, 1] > 30]  # [0,1]
ig_age = information_gain(y, left_y2, right_y2)

print(f'IG(income<=75): {ig_income:.3f}')
print(f'IG(age<=30):    {ig_age:.3f}')
print('Best split:', 'income' if ig_income > ig_age else 'age')

Using Gini in GridSearchCV

When tuning a decision tree with GridSearchCV, you can include the criterion parameter (gini vs entropy) in the parameter grid to let cross-validation choose the better option for your specific dataset. Combine this with max_depth, min_samples_split, and min_samples_leaf in the same search. Searching over criterion adds minimal computational cost — only two extra configurations per combination — and occasionally yields a meaningful improvement when class distributions are highly skewed or the dataset has many similar-quality splits.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [3, 5, 7, None],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best criterion:', grid.best_params_['criterion'])
print('Best depth:', grid.best_params_['max_depth'])
print('Best CV accuracy:', grid.best_score_.round(4))

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Gini impurity measures how mixed classes are in a node (formula: 1 - sum(p_i^2)), information gain measures how much a split reduces impurity (parent impurity minus weighted child impurity), and the tree always picks the split that maximises information gain across all features and thresholds. Next up we explore controlling tree depth to prevent overfitting.

Frequently asked questions

Is the “Gini Impurity and Information Gain” lesson free?

Yes — the full text of “Gini Impurity and Information Gain” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Gini Impurity and Information Gain”?

Learners will calculate Gini impurity and entropy for sample splits, understand why the tree picks the split that maximises information gain. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Gini Impurity and Information Gain” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Building a Tree: Splits, Nodes, and Leaves
  2. Gini Impurity and Information Gain
  3. Controlling Tree Depth to Prevent Overfitting
  4. Visualising and Interpreting Decision Trees
← Back to Machine Learning Academy