R Academy · Lesson

Decision Trees: Foundation of Ensembles

Build and visualize decision trees with rpart and understand bias-variance tradeoff.

Decision Trees: Foundation of Ensembles is a free R Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

How Decision Trees Split

A decision tree recursively partitions the feature space into rectangular regions. At each node, the algorithm searches over all features and all possible split points to find the split that best separates the target variable. The result is a tree of if-else rules.

library(rpart)

# Fit a classification tree
tree <- rpart(
  Species ~ .,
  data   = iris,
  method = 'class'  # use 'anova' for regression
)

print(tree)

GINI vs Entropy Split Criteria

The split criterion measures impurity of a node. Gini impurity measures the probability of misclassifying a randomly chosen element. Entropy (information gain) measures the reduction in information disorder. Both usually give similar trees; Gini is faster to compute and is the rpart default.

# Default: Gini impurity (parms = list(split = 'gini'))
tree_gini <- rpart(Species ~ ., data = iris, method = 'class')

# Using information gain (entropy)
tree_entropy <- rpart(
  Species ~ ., data = iris, method = 'class',
  parms = list(split = 'information')
)

cat('Gini root split:', tree_gini$frame$var[1])
cat('Entropy root split:', tree_entropy$frame$var[1])

printcp() — Complexity Table

printcp(tree) prints the complexity parameter (CP) table. Each row shows a tree size (number of splits), its relative error on training data, and its cross-validated error (xerror). The CP table is used to find the optimal pruning level.

tree <- rpart(medv ~ ., data = MASS::Boston, method = 'anova',
              control = rpart.control(minsplit = 5, cp = 0.001))

printcp(tree)

# Identify the CP with minimum cross-validated error
best_cp <- tree$cptable[
  which.min(tree$cptable[, 'xerror']),
  'CP'
]
cat('Best CP:', best_cp)

prune() — Trim the Tree

prune(tree, cp) prunes the tree back to the complexity level specified by cp. Pruning prevents overfitting by collapsing branches that provide little predictive value. The standard approach: find the CP that minimises CV error, then prune.

best_cp <- tree$cptable[
  which.min(tree$cptable[, 'xerror']), 'CP'
]

pruned_tree <- prune(tree, cp = best_cp)

cat('Original tree nodes:', nrow(tree$frame))
cat('Pruned tree nodes:', nrow(pruned_tree$frame))

rpart.plot() — Visualise the Tree

rpart.plot(tree) from the rpart.plot package produces a clean, coloured visualisation of the decision tree. Each internal node shows the split rule; each leaf shows the predicted class and the proportion of training samples.

library(rpart.plot)

tree <- rpart(Species ~ ., data = iris, method = 'class')
pruned <- prune(tree, cp = 0.02)

rpart.plot(
  pruned,
  type   = 4,     # split labels on branches
  extra  = 104,   # show class + probability
  fallen.leaves = TRUE
)

Bias-Variance Tradeoff

A deep, unpruned tree has low bias (fits training data almost perfectly) but high variance (small changes in data produce very different trees). A shallow or pruned tree has higher bias but lower variance. The optimal tree balances these two sources of error.

Ensemble methods like random forests and boosting attack this tradeoff directly.

# Deep tree = low bias, high variance (overfits)
deep_tree <- rpart(medv ~ ., data = MASS::Boston,
                   control = rpart.control(minsplit = 2, cp = 0))

# Shallow tree = high bias, low variance (underfits)
shallow_tree <- rpart(medv ~ ., data = MASS::Boston,
                      control = rpart.control(maxdepth = 2))

cat('Deep nodes:', nrow(deep_tree$frame))
cat('Shallow nodes:', nrow(shallow_tree$frame))

Overfitting a Decision Tree

An unpruned tree can achieve zero training error by memorising every training example. When the same tree is evaluated on unseen data, performance collapses. This is the canonical example of overfitting in supervised learning.

set.seed(42)
train_idx <- sample(nrow(MASS::Boston), 400)
train_bos <- MASS::Boston[train_idx, ]
test_bos  <- MASS::Boston[-train_idx, ]

# Fully grown tree
full <- rpart(medv ~ ., data = train_bos,
              control = rpart.control(cp = 0, minsplit = 2))

train_pred <- predict(full, train_bos)
test_pred  <- predict(full, test_bos)

cat('Train RMSE:', sqrt(mean((train_pred - train_bos$medv)^2)))
cat('Test  RMSE:', sqrt(mean((test_pred  - test_bos$medv)^2)))

Variable Importance from rpart

rpart records variable.importance for each predictor: the total improvement in the split criterion attributable to that variable across all splits. This gives a quick indication of which features drive the model's decisions.

tree <- rpart(medv ~ ., data = MASS::Boston, method = 'anova')

# Variable importance (sorted)
imp <- sort(tree$variable.importance, decreasing = TRUE)
print(imp)

# Quick barplot
barplot(imp, las = 2, main = 'Variable Importance',
        col = 'steelblue', cex.names = 0.8)

From Trees to Ensembles

A single decision tree is unstable: resampling the data produces very different trees. Ensemble methods exploit this instability:

Bagging / Random Forests: Average many trees on bootstrapped samples.
Boosting: Build trees sequentially, each correcting the previous tree's errors.
Both reduce variance while maintaining the expressive power of trees.

# Demonstrating instability of a single tree
set.seed(1); t1 <- rpart(medv ~ ., data = MASS::Boston[sample(506, 400), ])
set.seed(2); t2 <- rpart(medv ~ ., data = MASS::Boston[sample(506, 400), ])

# Root split may differ between trees
cat('Tree 1 root split:', t1$frame$var[1])
cat('Tree 2 root split:', t2$frame$var[1])

rpart Control Parameters

rpart.control() governs how the tree grows. Key parameters: cp (complexity penalty), minsplit (minimum observations to attempt a split), minbucket (minimum leaf size), and maxdepth. Understanding these is essential for tuning tree-based models.

ctrl <- rpart.control(
  cp        = 0.005,  # complexity penalty
  minsplit  = 20,     # min obs to try a split
  minbucket = 7,      # min obs in any leaf
  maxdepth  = 10      # max tree depth
)

tree <- rpart(medv ~ ., data = MASS::Boston,
              method = 'anova', control = ctrl)

printcp(tree)

Evaluating Tree Performance

After pruning, evaluate the tree on the held-out test set. For regression, compute RMSE and R-squared; for classification, compute accuracy and the confusion matrix. Compare these metrics to benchmark models to understand the value a single tree provides.

pruned_tree <- prune(tree, cp = best_cp)

test_pred <- predict(pruned_tree, newdata = test_bos)

rmse <- sqrt(mean((test_pred - test_bos$medv)^2))
ss_res <- sum((test_pred - test_bos$medv)^2)
ss_tot <- sum((test_bos$medv - mean(test_bos$medv))^2)
r2 <- 1 - ss_res / ss_tot

cat('RMSE:', round(rmse, 3))
cat('R2:', round(r2, 3))

Quick Check

Which statement best describes the purpose of calling prune(tree, cp = best_cp)?

Decision Trees Recap

Key takeaways from Decision Trees — Foundation of Ensembles:

Trees recursively partition feature space; splits use Gini or entropy criteria.
rpart(y ~ ., data, method) fits the tree; printcp() shows the complexity table.
Find the CP with minimum cross-validated error, then prune(tree, cp).
rpart.plot() visualises the tree structure.
Deep trees overfit (low bias, high variance); shallow trees underfit.
tree$variable.importance ranks predictors by their total split improvement.
Ensemble methods (random forests, boosting) overcome single-tree instability.

# Standard rpart workflow
tree <- rpart(y ~ ., data = train, method = 'anova',
              control = rpart.control(cp = 0.001))

best_cp <- tree$cptable[which.min(tree$cptable[, 'xerror']), 'CP']
pruned  <- prune(tree, cp = best_cp)

test_pred <- predict(pruned, newdata = test)
rmse <- sqrt(mean((test_pred - test$y)^2))
cat('Pruned Tree RMSE:', rmse)

Frequently asked questions

Is the “Decision Trees: Foundation of Ensembles” lesson free?

Yes — the full text of “Decision Trees: Foundation of Ensembles” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.

What will I learn in “Decision Trees: Foundation of Ensembles”?

Build and visualize decision trees with rpart and understand bias-variance tradeoff. You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start R Academy?

No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Decision Trees: Foundation of Ensembles” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this R Academy lesson?

Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Decision Trees: Foundation of Ensembles
Random Forests with ranger
Gradient Boosting with xgboost
Feature Importance and Model Interpretation

← Back to R Academy