Decision Trees: Foundation of Ensembles
Build and visualize decision trees with rpart and understand bias-variance tradeoff.
How Decision Trees Split
A decision tree recursively partitions the feature space into rectangular regions. At each node, the algorithm searches over all features and all possible split points to find the split that best separates the target variable. The result is a tree of if-else rules.
library(rpart)
# Fit a classification tree
tree <- rpart(
Species ~ .,
data = iris,
method = 'class' # use 'anova' for regression
)
print(tree)GINI vs Entropy Split Criteria
The split criterion measures impurity of a node. Gini impurity measures the probability of misclassifying a randomly chosen element. Entropy (information gain) measures the reduction in information disorder. Both usually give similar trees; Gini is faster to compute and is the rpart default.
# Default: Gini impurity (parms = list(split = 'gini'))
tree_gini <- rpart(Species ~ ., data = iris, method = 'class')
# Using information gain (entropy)
tree_entropy <- rpart(
Species ~ ., data = iris, method = 'class',
parms = list(split = 'information')
)
cat('Gini root split:', tree_gini$frame$var[1])
cat('Entropy root split:', tree_entropy$frame$var[1])All lessons in this course
- Decision Trees: Foundation of Ensembles
- Random Forests with ranger
- Gradient Boosting with xgboost
- Feature Importance and Model Interpretation