Machine Learning Academy · 강의

지니 불순도와 정보 이득

표본 분할의 지니 불순도와 엔트로피를 계산하고, 트리가 정보 이득을 최대화하는 분할을 선택하는 이유를 이해합니다.

지니 불순도와 정보 이득은(는) CoddyKit의 무료 Machine Learning Academy 강의입니다. 이것은 4개 중 2번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Machine Learning Academy 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Machine Learning Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

The Splitting Problem: Which Feature to Ask About?

When building a decision tree, at each node we must choose which feature and which threshold produces the most useful split. The goal is to create child nodes where samples are as pure as possible — ideally each child contains only one class. We need a mathematical measure of impurity that tells us how mixed the classes are in a node. Lower impurity is better: a node with all samples belonging to one class has zero impurity (perfect purity). Two widely-used impurity measures are Gini impurity and entropy.

# Impurity measures how mixed the classes are in a node
# Perfect purity: all samples belong to one class -> impurity = 0
# Maximum impurity: classes are equally distributed

import numpy as np

# Node A: all class 0 -> pure
node_a = [0, 0, 0, 0]  # impurity = 0

# Node B: 50/50 mix -> maximally impure
node_b = [0, 0, 1, 1]  # impurity = maximum

# Node C: mostly one class
node_c = [0, 0, 0, 1]  # impurity = low

for name, node in [('A', node_a), ('B', node_b), ('C', node_c)]:
    print(f'Node {name}: classes = {node}')

Gini Impurity: The Default Criterion

Gini impurity measures the probability that a randomly chosen sample from a node would be incorrectly labelled if labelled randomly according to the class distribution in that node. The formula is: Gini = 1 - sum(p_i^2) where p_i is the proportion of class i. Gini ranges from 0 (pure) to 0.5 (two-class equal split). For K classes, the maximum is 1 - 1/K. Gini impurity is the default criterion in scikit-learn's DecisionTreeClassifier because it is computationally efficient (no logarithms).

import numpy as np

def gini_impurity(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    return 1 - np.sum(probabilities ** 2)

# Pure node
print('Pure [0,0,0,0]:', gini_impurity([0,0,0,0]))       # 0.0

# 50/50 split
print('50/50 [0,0,1,1]:', gini_impurity([0,0,1,1]))      # 0.5

# 75/25 split
print('75/25 [0,0,0,1]:', gini_impurity([0,0,0,1]))      # 0.375

# Three classes equal
print('3-class equal:', gini_impurity([0,1,2,0,1,2]))    # ~0.667

Entropy and Information Theory

Entropy is borrowed from information theory: it measures the uncertainty or information content of a distribution. Formula: H = -sum(p_i * log2(p_i)). A pure node has entropy 0 (no uncertainty). A 50/50 split has entropy 1 (one bit of uncertainty — you need one question to determine the class). Entropy and Gini produce very similar trees in practice. Entropy is slightly slower to compute (requires log) but may give better splits when class distributions are skewed. Use criterion='entropy' in scikit-learn to switch.

import numpy as np

def entropy(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    # Avoid log(0) by filtering zero probabilities
    probs = probabilities[probabilities > 0]
    return -np.sum(probs * np.log2(probs))

print('Pure [0,0,0,0]:', entropy([0,0,0,0]))         # 0.0
print('50/50 [0,0,1,1]:', entropy([0,0,1,1]))        # 1.0 (1 bit)
print('75/25 [0,0,0,1]:', entropy([0,0,0,1]).round(3))  # 0.811
print('3-class equal:', entropy([0,1,2,0,1,2]).round(3)) # 1.585

Information Gain: The Split Quality Metric

Information gain measures how much a split reduces impurity. It is computed as the impurity of the parent node minus the weighted average impurity of the child nodes: IG = impurity(parent) - (N_left/N * impurity(left) + N_right/N * impurity(right)). The best split maximises information gain: it produces children that are as pure as possible, weighted by their size (so larger children count more). The tree builder evaluates information gain for every feature and every threshold, then picks the combination with the highest gain.

import numpy as np

def gini_impurity(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

def information_gain(y_parent, y_left, y_right):
    n = len(y_parent)
    n_l, n_r = len(y_left), len(y_right)
    parent_impurity = gini_impurity(y_parent)
    weighted_child  = (n_l/n)*gini_impurity(y_left) + (n_r/n)*gini_impurity(y_right)
    return parent_impurity - weighted_child

y_parent = [0,0,0,1,1,1]  # 50/50 parent
y_left   = [0,0,0]         # pure left
y_right  = [1,1,1]         # pure right

print('IG:', information_gain(y_parent, y_left, y_right))  # 0.5 (perfect split)

Evaluating Multiple Splits

To find the best split, the algorithm evaluates all candidate feature-threshold combinations and picks the one with the highest information gain. For a dataset with N samples and d features, the tree evaluates up to N-1 thresholds per feature (midpoints between consecutive unique values), giving O(N * d) splits to evaluate per node. Here is a simplified example showing how different thresholds produce different information gains on the same feature.

import numpy as np

X_feature = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 0, 1, 1, 1])

best_threshold, best_ig = None, -1

for threshold in [1.5, 2.5, 3.5, 4.5, 5.5]:
    left_mask  = X_feature <= threshold
    right_mask = ~left_mask
    y_l, y_r = y[left_mask], y[right_mask]
    
    _, cnt_p = np.unique(y, return_counts=True)
    _, cnt_l = np.unique(y_l, return_counts=True) if len(y_l) else (None, [1])
    _, cnt_r = np.unique(y_r, return_counts=True) if len(y_r) else (None, [1])
    
    ig = information_gain(y, y_l, y_r)
    print(f'Threshold {threshold}: IG = {ig:.3f}')
    if ig > best_ig:
        best_ig, best_threshold = ig, threshold

print('Best threshold:', best_threshold, 'with IG:', best_ig)

Gini vs Entropy: Practical Difference

Gini impurity and entropy produce nearly identical trees most of the time. The key differences are subtle: entropy tends to produce more balanced trees (it penalises imbalanced splits more heavily due to the logarithm), while Gini tends to isolate the most frequent class in one branch. Computationally, Gini is faster because it avoids the logarithm computation. In practice, the choice between them is a hyperparameter to tune — try both with cross-validation and pick the one that performs better on your specific dataset.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for criterion in ['gini', 'entropy']:
    tree = DecisionTreeClassifier(criterion=criterion, max_depth=5, random_state=42)
    score = cross_val_score(tree, X, y, cv=10).mean()
    print(f'criterion={criterion}: CV accuracy = {score:.3f}')

# Usually within 0.5% of each other -- not the critical choice

Weighted Gini for Multi-Class Problems

Gini impurity extends naturally to multi-class problems with no modification: Gini = 1 - sum(p_i^2) works for any number of classes. Information gain computation is also unchanged — weighted impurity of children minus parent impurity. For a 3-class problem, a perfectly pure leaf (all class 2, for example) has Gini 0. A node with equal proportions of 3 classes has maximum Gini of 2/3. Decision trees are one of the few algorithms that handle multi-class problems natively without any modification — unlike logistic regression, which requires one-vs-rest or softmax extensions.

import numpy as np

def gini_multiclass(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

# 3-class examples
print('Pure [0,0,0]:', gini_multiclass([0,0,0]))               # 0.0
print('Equal [0,1,2]:', gini_multiclass([0,1,2]).round(3))    # 0.667
print('2 dominant [0,0,1,2]:', gini_multiclass([0,0,1,2]).round(3))  # 0.625

# Max Gini for K classes = 1 - 1/K
for K in [2, 3, 4, 5]:
    print(f'Max Gini for {K} classes: {1 - 1/K:.3f}')

Impurity Reduction Inside scikit-learn

Inside scikit-learn, at each node the tree stores the impurity before the split and the impurity of each child. The difference, weighted by sample count, is the impurity reduction (information gain). This value is summed per feature across all nodes and normalised to compute feature_importances_ — the total impurity reduction attributed to each feature. Features that appear near the root and handle many samples tend to have the highest importance because each split affects a large fraction of the data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
import numpy as np

X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=42)
tree.fit(X, y)

# Impurity at root and children
print('Root impurity (Gini):', tree.tree_.impurity[0].round(4))
print('Left child impurity:', tree.tree_.impurity[1].round(4))
print('Right child impurity:', tree.tree_.impurity[2].round(4))

# Feature importances = total weighted impurity reduction per feature
print('Feature importances:', tree.feature_importances_.round(3))

The Role of min_impurity_decrease

By default, the tree splits nodes as long as there is any reduction in impurity and the node has enough samples. The min_impurity_decrease parameter adds a minimum threshold: a split is only created if it reduces impurity by at least this amount. This prevents the tree from making trivially small splits that memorise noise. Setting min_impurity_decrease=0.01 means the tree only splits when the information gain exceeds 0.01. This is a useful regularisation alternative to controlling depth directly.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_ig in [0.0, 0.001, 0.005, 0.01, 0.05]:
    tree = DecisionTreeClassifier(
        min_impurity_decrease=min_ig,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=5).mean()
    depth = tree.fit(X, y).get_depth()
    print(f'min_impurity_decrease={min_ig}: depth={depth}, CV acc={score:.3f}')

Comparing Splits Across Features

A worked example showing how the tree picks the best feature and threshold. Given two features and one split threshold each, the tree computes information gain for all options and picks the winner. This illustrates why trees naturally perform implicit feature selection: features that never produce high information gain at any threshold will never be chosen as split features, effectively being ignored. This makes decision trees robust to irrelevant features, unlike KNN which is harmed by them.

import numpy as np

# Toy dataset: X[:,0]=income, X[:,1]=age; y=churn
X = np.array([[100, 25], [120, 30], [40, 22], [50, 28], [90, 35], [30, 40]])
y = np.array([0, 0, 1, 1, 0, 1])

# Try splitting on income at 75
left_y = y[X[:, 0] <= 75]   # [1,1,1]
right_y = y[X[:, 0] > 75]   # [0,0,0]
ig_income = information_gain(y, left_y, right_y)

# Try splitting on age at 30
left_y2 = y[X[:, 1] <= 30]  # [0,0,1,1]
right_y2 = y[X[:, 1] > 30]  # [0,1]
ig_age = information_gain(y, left_y2, right_y2)

print(f'IG(income<=75): {ig_income:.3f}')
print(f'IG(age<=30):    {ig_age:.3f}')
print('Best split:', 'income' if ig_income > ig_age else 'age')

Using Gini in GridSearchCV

When tuning a decision tree with GridSearchCV, you can include the criterion parameter (gini vs entropy) in the parameter grid to let cross-validation choose the better option for your specific dataset. Combine this with max_depth, min_samples_split, and min_samples_leaf in the same search. Searching over criterion adds minimal computational cost — only two extra configurations per combination — and occasionally yields a meaningful improvement when class distributions are highly skewed or the dataset has many similar-quality splits.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [3, 5, 7, None],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best criterion:', grid.best_params_['criterion'])
print('Best depth:', grid.best_params_['max_depth'])
print('Best CV accuracy:', grid.best_score_.round(4))

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Gini impurity measures how mixed classes are in a node (formula: 1 - sum(p_i^2)), information gain measures how much a split reduces impurity (parent impurity minus weighted child impurity), and the tree always picks the split that maximises information gain across all features and thresholds. Next up we explore controlling tree depth to prevent overfitting.

자주 묻는 질문

“지니 불순도와 정보 이득” 강의는 무료인가요?

네 — “지니 불순도와 정보 이득” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Machine Learning Academy 강의 전체를 잠금 해제할 수 있습니다. Machine Learning Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

“지니 불순도와 정보 이득”에서 뭘 배우나요?

표본 분할의 지니 불순도와 엔트로피를 계산하고, 트리가 정보 이득을 최대화하는 분할을 선택하는 이유를 이해합니다. 브라우저에서 직접 실행하는 실습 코드로 Machine Learning Academy을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

Machine Learning Academy을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 Machine Learning Academy은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 2번째 강의입니다.

“지니 불순도와 정보 이득” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 Machine Learning Academy 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 Machine Learning Academy 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← Machine Learning Academy(으)로 돌아가기