Feature Importance and Model Interpretation
Extract and visualize variable importance and SHAP values from ensemble models.
Why Model Interpretation Matters
Accurate models that nobody trusts are useless in practice. Model interpretation answers two questions: Which features matter most globally? (global importance) and Why did the model make this specific prediction? (local explanation). Both XGBoost and ranger provide built-in tools for global importance, while SHAP values enable local explanations.
# The interpretation toolkit we will use:
# - xgb.importance() XGBoost global importance
# - xgb.plot.importance() Visualise XGBoost importance
# - ranger importance Random forest importance
# - shapviz package SHAP value computation + plots
cat('Packages needed: xgboost, ranger, shapviz')xgb.importance() — Three Metrics
xgb.importance(model) returns a data frame with three importance metrics for each feature:
- Gain: average improvement in loss from splits on this feature (most informative).
- Cover: average number of observations affected by splits on this feature.
- Frequency: proportion of splits that used this feature.
library(xgboost)
library(MASS)
X <- as.matrix(Boston[, -14])
y <- Boston[, 14]
dtrain <- xgb.DMatrix(data = X, label = y)
params <- list(objective = 'reg:squarederror', eta = 0.1,
max_depth = 5, subsample = 0.8)
model <- xgboost(data = dtrain, params = params,
nrounds = 100, verbose = 0)
imp <- xgb.importance(model = model, feature_names = colnames(X))
print(head(imp))All lessons in this course
- Decision Trees: Foundation of Ensembles
- Random Forests with ranger
- Gradient Boosting with xgboost
- Feature Importance and Model Interpretation