0Pricing
Machine Learning Academy · Ders

Gini Safsızlığı ve Bilgi Kazancı

Örnek bölmeleri için Gini safsızlığını ve entropiyi hesaplayın; ağacın bilgi kazancını en üst düzeye çıkaran bölmeyi neden seçtiğini anlayın.

Gini Safsızlığı ve Bilgi Kazancı, CoddyKit'te ücretsiz bir Machine Learning Academy dersidir. Bu, 4 dersinin 2. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, Machine Learning Academy öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. Machine Learning Academy kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

The Splitting Problem: Which Feature to Ask About?

When building a decision tree, at each node we must choose which feature and which threshold produces the most useful split. The goal is to create child nodes where samples are as pure as possible — ideally each child contains only one class. We need a mathematical measure of impurity that tells us how mixed the classes are in a node. Lower impurity is better: a node with all samples belonging to one class has zero impurity (perfect purity). Two widely-used impurity measures are Gini impurity and entropy.

# Impurity measures how mixed the classes are in a node
# Perfect purity: all samples belong to one class -> impurity = 0
# Maximum impurity: classes are equally distributed

import numpy as np

# Node A: all class 0 -> pure
node_a = [0, 0, 0, 0]  # impurity = 0

# Node B: 50/50 mix -> maximally impure
node_b = [0, 0, 1, 1]  # impurity = maximum

# Node C: mostly one class
node_c = [0, 0, 0, 1]  # impurity = low

for name, node in [('A', node_a), ('B', node_b), ('C', node_c)]:
    print(f'Node {name}: classes = {node}')

Gini Impurity: The Default Criterion

Gini impurity measures the probability that a randomly chosen sample from a node would be incorrectly labelled if labelled randomly according to the class distribution in that node. The formula is: Gini = 1 - sum(p_i^2) where p_i is the proportion of class i. Gini ranges from 0 (pure) to 0.5 (two-class equal split). For K classes, the maximum is 1 - 1/K. Gini impurity is the default criterion in scikit-learn's DecisionTreeClassifier because it is computationally efficient (no logarithms).

import numpy as np

def gini_impurity(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    return 1 - np.sum(probabilities ** 2)

# Pure node
print('Pure [0,0,0,0]:', gini_impurity([0,0,0,0]))       # 0.0

# 50/50 split
print('50/50 [0,0,1,1]:', gini_impurity([0,0,1,1]))      # 0.5

# 75/25 split
print('75/25 [0,0,0,1]:', gini_impurity([0,0,0,1]))      # 0.375

# Three classes equal
print('3-class equal:', gini_impurity([0,1,2,0,1,2]))    # ~0.667

Entropy and Information Theory

Entropy is borrowed from information theory: it measures the uncertainty or information content of a distribution. Formula: H = -sum(p_i * log2(p_i)). A pure node has entropy 0 (no uncertainty). A 50/50 split has entropy 1 (one bit of uncertainty — you need one question to determine the class). Entropy and Gini produce very similar trees in practice. Entropy is slightly slower to compute (requires log) but may give better splits when class distributions are skewed. Use criterion='entropy' in scikit-learn to switch.

import numpy as np

def entropy(y):
    classes, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    # Avoid log(0) by filtering zero probabilities
    probs = probabilities[probabilities > 0]
    return -np.sum(probs * np.log2(probs))

print('Pure [0,0,0,0]:', entropy([0,0,0,0]))         # 0.0
print('50/50 [0,0,1,1]:', entropy([0,0,1,1]))        # 1.0 (1 bit)
print('75/25 [0,0,0,1]:', entropy([0,0,0,1]).round(3))  # 0.811
print('3-class equal:', entropy([0,1,2,0,1,2]).round(3)) # 1.585

Information Gain: The Split Quality Metric

Information gain measures how much a split reduces impurity. It is computed as the impurity of the parent node minus the weighted average impurity of the child nodes: IG = impurity(parent) - (N_left/N * impurity(left) + N_right/N * impurity(right)). The best split maximises information gain: it produces children that are as pure as possible, weighted by their size (so larger children count more). The tree builder evaluates information gain for every feature and every threshold, then picks the combination with the highest gain.

import numpy as np

def gini_impurity(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

def information_gain(y_parent, y_left, y_right):
    n = len(y_parent)
    n_l, n_r = len(y_left), len(y_right)
    parent_impurity = gini_impurity(y_parent)
    weighted_child  = (n_l/n)*gini_impurity(y_left) + (n_r/n)*gini_impurity(y_right)
    return parent_impurity - weighted_child

y_parent = [0,0,0,1,1,1]  # 50/50 parent
y_left   = [0,0,0]         # pure left
y_right  = [1,1,1]         # pure right

print('IG:', information_gain(y_parent, y_left, y_right))  # 0.5 (perfect split)

Evaluating Multiple Splits

To find the best split, the algorithm evaluates all candidate feature-threshold combinations and picks the one with the highest information gain. For a dataset with N samples and d features, the tree evaluates up to N-1 thresholds per feature (midpoints between consecutive unique values), giving O(N * d) splits to evaluate per node. Here is a simplified example showing how different thresholds produce different information gains on the same feature.

import numpy as np

X_feature = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 0, 1, 1, 1])

best_threshold, best_ig = None, -1

for threshold in [1.5, 2.5, 3.5, 4.5, 5.5]:
    left_mask  = X_feature <= threshold
    right_mask = ~left_mask
    y_l, y_r = y[left_mask], y[right_mask]
    
    _, cnt_p = np.unique(y, return_counts=True)
    _, cnt_l = np.unique(y_l, return_counts=True) if len(y_l) else (None, [1])
    _, cnt_r = np.unique(y_r, return_counts=True) if len(y_r) else (None, [1])
    
    ig = information_gain(y, y_l, y_r)
    print(f'Threshold {threshold}: IG = {ig:.3f}')
    if ig > best_ig:
        best_ig, best_threshold = ig, threshold

print('Best threshold:', best_threshold, 'with IG:', best_ig)

Gini vs Entropy: Practical Difference

Gini impurity and entropy produce nearly identical trees most of the time. The key differences are subtle: entropy tends to produce more balanced trees (it penalises imbalanced splits more heavily due to the logarithm), while Gini tends to isolate the most frequent class in one branch. Computationally, Gini is faster because it avoids the logarithm computation. In practice, the choice between them is a hyperparameter to tune — try both with cross-validation and pick the one that performs better on your specific dataset.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for criterion in ['gini', 'entropy']:
    tree = DecisionTreeClassifier(criterion=criterion, max_depth=5, random_state=42)
    score = cross_val_score(tree, X, y, cv=10).mean()
    print(f'criterion={criterion}: CV accuracy = {score:.3f}')

# Usually within 0.5% of each other -- not the critical choice

Weighted Gini for Multi-Class Problems

Gini impurity extends naturally to multi-class problems with no modification: Gini = 1 - sum(p_i^2) works for any number of classes. Information gain computation is also unchanged — weighted impurity of children minus parent impurity. For a 3-class problem, a perfectly pure leaf (all class 2, for example) has Gini 0. A node with equal proportions of 3 classes has maximum Gini of 2/3. Decision trees are one of the few algorithms that handle multi-class problems natively without any modification — unlike logistic regression, which requires one-vs-rest or softmax extensions.

import numpy as np

def gini_multiclass(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1 - np.sum(p**2)

# 3-class examples
print('Pure [0,0,0]:', gini_multiclass([0,0,0]))               # 0.0
print('Equal [0,1,2]:', gini_multiclass([0,1,2]).round(3))    # 0.667
print('2 dominant [0,0,1,2]:', gini_multiclass([0,0,1,2]).round(3))  # 0.625

# Max Gini for K classes = 1 - 1/K
for K in [2, 3, 4, 5]:
    print(f'Max Gini for {K} classes: {1 - 1/K:.3f}')

Impurity Reduction Inside scikit-learn

Inside scikit-learn, at each node the tree stores the impurity before the split and the impurity of each child. The difference, weighted by sample count, is the impurity reduction (information gain). This value is summed per feature across all nodes and normalised to compute feature_importances_ — the total impurity reduction attributed to each feature. Features that appear near the root and handle many samples tend to have the highest importance because each split affects a large fraction of the data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
import numpy as np

X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=42)
tree.fit(X, y)

# Impurity at root and children
print('Root impurity (Gini):', tree.tree_.impurity[0].round(4))
print('Left child impurity:', tree.tree_.impurity[1].round(4))
print('Right child impurity:', tree.tree_.impurity[2].round(4))

# Feature importances = total weighted impurity reduction per feature
print('Feature importances:', tree.feature_importances_.round(3))

The Role of min_impurity_decrease

By default, the tree splits nodes as long as there is any reduction in impurity and the node has enough samples. The min_impurity_decrease parameter adds a minimum threshold: a split is only created if it reduces impurity by at least this amount. This prevents the tree from making trivially small splits that memorise noise. Setting min_impurity_decrease=0.01 means the tree only splits when the information gain exceeds 0.01. This is a useful regularisation alternative to controlling depth directly.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

for min_ig in [0.0, 0.001, 0.005, 0.01, 0.05]:
    tree = DecisionTreeClassifier(
        min_impurity_decrease=min_ig,
        random_state=42
    )
    score = cross_val_score(tree, X, y, cv=5).mean()
    depth = tree.fit(X, y).get_depth()
    print(f'min_impurity_decrease={min_ig}: depth={depth}, CV acc={score:.3f}')

Comparing Splits Across Features

A worked example showing how the tree picks the best feature and threshold. Given two features and one split threshold each, the tree computes information gain for all options and picks the winner. This illustrates why trees naturally perform implicit feature selection: features that never produce high information gain at any threshold will never be chosen as split features, effectively being ignored. This makes decision trees robust to irrelevant features, unlike KNN which is harmed by them.

import numpy as np

# Toy dataset: X[:,0]=income, X[:,1]=age; y=churn
X = np.array([[100, 25], [120, 30], [40, 22], [50, 28], [90, 35], [30, 40]])
y = np.array([0, 0, 1, 1, 0, 1])

# Try splitting on income at 75
left_y = y[X[:, 0] <= 75]   # [1,1,1]
right_y = y[X[:, 0] > 75]   # [0,0,0]
ig_income = information_gain(y, left_y, right_y)

# Try splitting on age at 30
left_y2 = y[X[:, 1] <= 30]  # [0,0,1,1]
right_y2 = y[X[:, 1] > 30]  # [0,1]
ig_age = information_gain(y, left_y2, right_y2)

print(f'IG(income<=75): {ig_income:.3f}')
print(f'IG(age<=30):    {ig_age:.3f}')
print('Best split:', 'income' if ig_income > ig_age else 'age')

Using Gini in GridSearchCV

When tuning a decision tree with GridSearchCV, you can include the criterion parameter (gini vs entropy) in the parameter grid to let cross-validation choose the better option for your specific dataset. Combine this with max_depth, min_samples_split, and min_samples_leaf in the same search. Searching over criterion adds minimal computational cost — only two extra configurations per combination — and occasionally yields a meaningful improvement when class distributions are highly skewed or the dataset has many similar-quality splits.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [3, 5, 7, None],
    'min_samples_leaf': [1, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=10, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print('Best criterion:', grid.best_params_['criterion'])
print('Best depth:', grid.best_params_['max_depth'])
print('Best CV accuracy:', grid.best_score_.round(4))

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Gini impurity measures how mixed classes are in a node (formula: 1 - sum(p_i^2)), information gain measures how much a split reduces impurity (parent impurity minus weighted child impurity), and the tree always picks the split that maximises information gain across all features and thresholds. Next up we explore controlling tree depth to prevent overfitting.

Sıkça Sorulan Sorular

“Gini Safsızlığı ve Bilgi Kazancı” dersi ücretsiz mi?

Evet — “Gini Safsızlığı ve Bilgi Kazancı” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve Machine Learning Academy kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. Machine Learning Academy kursu toplamda 4 dersten oluşur.

“Gini Safsızlığı ve Bilgi Kazancı” dersinde ne öğreneceğim?

Örnek bölmeleri için Gini safsızlığını ve entropiyi hesaplayın; ağacın bilgi kazancını en üst düzeye çıkaran bölmeyi neden seçtiğini anlayın. Machine Learning Academy ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

Machine Learning Academy öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te Machine Learning Academy, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 2. dersidir.

“Gini Safsızlığı ve Bilgi Kazancı” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu Machine Learning Academy dersinde kod yazıp çalıştırabilir miyim?

Evet. Her Machine Learning Academy dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

  1. Ağaç Oluşturma: Bölmeler, Düğümler ve Yapraklar
  2. Gini Safsızlığı ve Bilgi Kazancı
  3. Aşırı Uydurmayı Önlemek için Ağaç Derinliğini Denetleme
  4. Karar Ağaçlarını Görselleştirme ve Yorumlama
← Machine Learning Academy Sayfasına Dön