0Pricing
R Academy · Lesson

Gradient Boosting with xgboost

Configure xgboost parameters, early stopping, and learning rate schedules.

Gradient Boosting with xgboost is a free R Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Gradient Boosting?

Gradient boosting builds an ensemble sequentially. Each new tree corrects the residual errors of the previous ensemble by fitting to the negative gradient of the loss function. Unlike random forests (parallel trees), boosting builds trees one at a time, each learning from the mistakes of the last.

library(xgboost)

# XGBoost expects data in a special matrix format
# We'll build a simple example step by step
cat('XGBoost version:', packageVersion('xgboost'))

# Key concept: each tree reduces the ensemble error
# Final prediction = sum of all tree predictions

xgb.DMatrix()

xgb.DMatrix(data, label) is XGBoost's optimised internal data format. It stores the feature matrix and label vector together, enabling fast memory-efficient computation. Always convert your data to DMatrix before training.

library(xgboost)
library(MASS)

X_train <- as.matrix(Boston[1:400, -14])  # features
y_train <- Boston[1:400, 14]              # medv (target)

X_test  <- as.matrix(Boston[401:506, -14])
y_test  <- Boston[401:506, 14]

dtrain <- xgb.DMatrix(data = X_train, label = y_train)
dtest  <- xgb.DMatrix(data = X_test,  label = y_test)

cat('DMatrix rows:', nrow(dtrain))

xgboost() — Basic Training

xgboost(data, nrounds, eta, max_depth, objective) trains the model. Key parameters: eta (learning rate, smaller = more robust but slower), max_depth (tree depth, controls model complexity), nrounds (number of trees).

params <- list(
  objective  = 'reg:squarederror',
  eta        = 0.1,     # learning rate
  max_depth  = 6,       # tree depth
  subsample  = 0.8,     # row subsampling
  colsample_bytree = 0.8  # column subsampling
)

set.seed(42)
model <- xgboost(
  data    = dtrain,
  params  = params,
  nrounds = 100,
  verbose = 0
)

cat('Model trained with', model$niter, 'rounds')

Watchlist — Monitor Validation Loss

The watchlist argument accepts a named list of DMatrix objects. XGBoost evaluates and prints the loss on each listed dataset after every boosting round. Use it to track training vs validation loss and detect when overfitting begins.

watchlist <- list(train = dtrain, eval = dtest)

model <- xgb.train(
  params    = params,
  data      = dtrain,
  nrounds   = 200,
  watchlist = watchlist,
  verbose   = 1
)

# The log shows train-rmse and eval-rmse per round
# Watch for eval-rmse increasing (overfitting signal)

early_stopping_rounds

early_stopping_rounds = 20 stops training if the validation metric has not improved for 20 consecutive rounds. This automatically finds the optimal number of trees, avoiding both underfitting and overfitting without exhaustive manual tuning.

model <- xgb.train(
  params              = params,
  data                = dtrain,
  nrounds             = 1000,      # max rounds
  watchlist           = list(train = dtrain, eval = dtest),
  early_stopping_rounds = 20,      # stop if no improvement
  print_every_n       = 50,
  verbose             = 1
)

cat('Best iteration:', model$best_iteration)
cat('Best eval RMSE:', model$best_score)

Binary Classification

For binary classification, use objective = 'binary:logistic', which outputs predicted probabilities. The label must be 0/1 numeric. Evaluate with AUC or log-loss using the eval_metric parameter.

# Binary classification example
library(MASS)
Pima <- MASS::Pima.tr

X_cl <- as.matrix(Pima[, -8])
y_cl <- as.numeric(Pima$type) - 1  # factor to 0/1

dt_cl <- xgb.DMatrix(data = X_cl, label = y_cl)

mod_cl <- xgboost(
  data       = dt_cl,
  objective  = 'binary:logistic',
  eval_metric = 'auc',
  eta        = 0.05,
  max_depth  = 4,
  nrounds    = 100,
  verbose    = 0
)

Predictions

predict(model, dtest) returns raw predicted values — probabilities for classification, or real-valued predictions for regression. Apply a 0.5 threshold to convert probabilities to class labels for classification tasks.

# Regression predictions
reg_preds <- predict(model, dtest)
rmse <- sqrt(mean((reg_preds - y_test)^2))
cat('Test RMSE:', round(rmse, 3))

# Classification predictions
prob_preds  <- predict(mod_cl, dt_cl)
class_preds <- ifelse(prob_preds > 0.5, 1, 0)
accuracy <- mean(class_preds == y_cl)
cat('Training Accuracy:', round(accuracy, 3))

xgb.cv() — Cross Validation

xgb.cv(params, data, nrounds, nfold) runs k-fold cross validation within XGBoost. This is faster than using rsample because XGBoost handles the folds internally. The output shows mean and SD of the metric per round.

cv_result <- xgb.cv(
  params    = params,
  data      = dtrain,
  nrounds   = 200,
  nfold     = 5,
  early_stopping_rounds = 15,
  print_every_n = 20,
  verbose   = 1
)

# Best nrounds from CV
best_nrounds <- cv_result$best_iteration
cat('Optimal nrounds:', best_nrounds)

# CV RMSE at best iteration
cat('CV RMSE:', cv_result$evaluation_log[best_nrounds, 'test_rmse_mean'][[1]])

Key Hyperparameters

The most impactful XGBoost hyperparameters:

  • eta: Learning rate (0.01-0.3). Lower = more trees needed, more robust.
  • max_depth: Tree depth (3-10). Larger = more complex, overfits faster.
  • subsample: Row fraction per tree (0.5-1.0). Reduces overfitting.
  • colsample_bytree: Feature fraction per tree (0.5-1.0).
  • lambda: L2 regularization on leaf weights.
  • alpha: L1 regularization on leaf weights.
params_tuned <- list(
  objective         = 'reg:squarederror',
  eta               = 0.05,
  max_depth         = 5,
  subsample         = 0.75,
  colsample_bytree  = 0.75,
  lambda            = 1.0,    # L2 regularization
  alpha             = 0.1,    # L1 regularization
  min_child_weight  = 3       # min samples in leaf
)

model_tuned <- xgboost(
  data = dtrain, params = params_tuned,
  nrounds = best_nrounds, verbose = 0
)

Saving and Loading Models

XGBoost models can be saved to disk in a binary format with xgb.save(model, 'model.xgb') and reloaded with xgb.load('model.xgb'). This is the recommended format for production deployment and ensures bit-exact reproducibility.

# Save model
xgb.save(model, '/tmp/xgb_boston.xgb')

# Reload and predict
loaded_model <- xgb.load('/tmp/xgb_boston.xgb')
new_preds <- predict(loaded_model, dtest)

# Verify predictions match
all.equal(reg_preds, new_preds)  # TRUE

Multi-class Classification

For multi-class problems, use objective = 'multi:softprob' and set num_class to the number of classes. Labels must be 0-indexed integers. The output is a matrix of probabilities for each class.

X_mc <- as.matrix(iris[, -5])
y_mc <- as.integer(iris$Species) - 1  # 0, 1, 2

dt_mc <- xgb.DMatrix(data = X_mc, label = y_mc)

params_mc <- list(
  objective  = 'multi:softprob',
  num_class  = 3,
  eta        = 0.1,
  max_depth  = 3
)

mod_mc <- xgboost(
  data = dt_mc, params = params_mc,
  nrounds = 50, verbose = 0
)

# Predictions: matrix of shape (n, num_class)
prob_matrix <- matrix(predict(mod_mc, dt_mc), ncol = 3, byrow = TRUE)
head(prob_matrix)

Quick Check

What is the purpose of setting early_stopping_rounds = 20 in xgb.train()?

XGBoost Recap

Key takeaways from Gradient Boosting with XGBoost:

  • xgb.DMatrix(data, label) creates XGBoost's optimised data format.
  • Key params: eta (learning rate), max_depth, subsample, colsample_bytree.
  • watchlist monitors train and validation loss per round.
  • early_stopping_rounds finds the optimal number of trees automatically.
  • xgb.cv() runs k-fold CV within XGBoost for fast hyperparameter search.
  • Use objective = 'binary:logistic' for binary, 'multi:softprob' for multi-class.
  • Save/load models with xgb.save() / xgb.load().
# Production XGBoost pipeline
dtrain <- xgb.DMatrix(data = X_train, label = y_train)
dtest  <- xgb.DMatrix(data = X_test,  label = y_test)

params <- list(objective = 'reg:squarederror',
               eta = 0.05, max_depth = 5, subsample = 0.8)

cv  <- xgb.cv(params, dtrain, nrounds = 500, nfold = 5,
              early_stopping_rounds = 20, verbose = 0)

model <- xgboost(data = dtrain, params = params,
                 nrounds = cv$best_iteration, verbose = 0)

preds <- predict(model, dtest)
cat('RMSE:', sqrt(mean((preds - y_test)^2)))

Frequently asked questions

Is the “Gradient Boosting with xgboost” lesson free?

Yes — the full text of “Gradient Boosting with xgboost” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.

What will I learn in “Gradient Boosting with xgboost”?

Configure xgboost parameters, early stopping, and learning rate schedules. You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start R Academy?

No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Gradient Boosting with xgboost” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this R Academy lesson?

Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Decision Trees: Foundation of Ensembles
  2. Random Forests with ranger
  3. Gradient Boosting with xgboost
  4. Feature Importance and Model Interpretation
← Back to R Academy