XGBoost: Regularisation, Early Stopping, and Feature Importance
Learners will train an XGBClassifier, enable early stopping on a validation set, and plot feature importance scores to identify the most predictive columns.
XGBoost: Regularisation, Early Stopping, and Feature Importance is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is XGBoost?
XGBoost (eXtreme Gradient Boosting) is a highly optimised gradient boosting library that dominated Kaggle competitions from 2014 onward. It improves on scikit-learn's GradientBoostingClassifier in three major ways: (1) built-in L1 and L2 regularisation on tree weights to reduce overfitting, (2) a second-order Taylor expansion of the loss for more accurate gradient estimates, and (3) a highly efficient approximate histogram-based split finding algorithm that scales to datasets with millions of rows.
Installing and Importing XGBoost
XGBoost is a standalone library installed separately from scikit-learn. It provides a sklearn-compatible API through XGBClassifier and XGBRegressor, so you can use it with cross_val_score, GridSearchCV, and Pipelines just like any scikit-learn estimator. The native XGBoost API uses xgb.DMatrix and xgb.train(), offering more fine-grained control over early stopping and custom objectives.
# Install: pip install xgboost
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1, max_depth=3,
use_label_encoder=False, eval_metric='logloss', random_state=42)
model.fit(X_train, y_train)
print('XGBoost test accuracy:', model.score(X_test, y_test))XGBoost Regularisation: lambda and alpha
XGBoost exposes two regularisation terms: reg_lambda (L2 penalty on leaf weights, default=1) and reg_alpha (L1 penalty on leaf weights, default=0). L2 regularisation shrinks leaf weights toward zero smoothly; L1 can set some leaf weights to exactly zero (sparse tree structure). Both reduce overfitting on noisy datasets. Additionally, min_child_weight requires a minimum sum of instance weights in a child node before a split is made, acting like a minimum-samples-per-leaf constraint.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
for lam in [0, 1, 5, 10]:
model = xgb.XGBClassifier(n_estimators=100, reg_lambda=lam, eval_metric='logloss',
random_state=42, verbosity=0)
score = cross_val_score(model, X, y, cv=5).mean()
print(f'reg_lambda={lam:3d}: CV accuracy={score:.4f}')Early Stopping: Stop When You Stop Improving
Early stopping monitors a validation metric after each boosting round and stops training when the metric has not improved for a specified number of rounds (early_stopping_rounds). This prevents overfitting and saves computation — you can safely set n_estimators very high (e.g., 1000) and let early stopping find the optimal number of rounds. The best iteration is stored in model.best_iteration and is used automatically for predictions.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=3,
eval_metric='logloss', verbosity=0, random_state=42)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
early_stopping_rounds=20,
verbose=False)
print('Best iteration:', model.best_iteration)
print('Test accuracy:', model.score(X_val, y_val))XGBoost Feature Importance
XGBoost provides three types of feature importance: 'weight' (number of times a feature is used in splits), 'gain' (average improvement in loss when a feature is used for splitting — usually the most informative), and 'cover' (average number of samples affected by splits on a feature). Access them via model.feature_importances_ (uses gain by default in the sklearn API) or model.get_booster().get_score(importance_type='gain').
import xgboost as xgb
import pandas as pd
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X, y = data.data, data.target
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(X, y)
importances = pd.Series(model.feature_importances_, index=data.feature_names)
print(importances.sort_values(ascending=False).head(5))Plotting Feature Importance
XGBoost includes a built-in plotting utility xgb.plot_importance(model) that creates a horizontal bar chart of feature importances. For more customisation, use the Series from model.feature_importances_ and plot with matplotlib. Feature importance from boosting is computed differently than from random forests — it reflects how much each feature contributed to reducing the loss across all trees, weighted by usage frequency or average gain.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
import matplotlib.pyplot as plt
data = load_breast_cancer()
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(data.data, data.target)
# xgb.plot_importance(model, max_num_features=10) # uncomment in Jupyter
# plt.show()
print('Top feature:', data.feature_names[model.feature_importances_.argmax()])Subsampling Parameters in XGBoost
XGBoost provides three subsampling parameters for additional regularisation: subsample (fraction of training rows used per tree, e.g. 0.8), colsample_bytree (fraction of features used per tree, e.g. 0.8), and colsample_bylevel (fraction of features per depth level). Together these introduce randomness similar to random forests' feature sub-sampling, reducing correlation between trees. Typical starting values: subsample=0.8, colsample_bytree=0.8.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
model = xgb.XGBClassifier(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
subsample=0.8,
colsample_bytree=0.8,
reg_lambda=2,
eval_metric='logloss',
random_state=42
)
print('XGBoost with subsampling CV:', cross_val_score(model, X, y, cv=5).mean().round(4))XGBoost for Regression
XGBRegressor uses the same engine but optimises a regression loss (squared error by default, or Tweedie, gamma, quantile, etc.). Early stopping with a regression metric (e.g., RMSE) works exactly the same way. XGBoost is particularly competitive on tabular regression tasks because it handles missing values natively (learns the best direction to send missing-value nodes during tree construction) and supports monotonicity constraints for domain-specific feature relationships.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
model = xgb.XGBRegressor(n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, eval_metric='rmse', random_state=42)
rmse = np.sqrt(-cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost Regression RMSE:', round(rmse, 4))XGBoost with Cross-Validation and GridSearch
Because XGBClassifier implements the scikit-learn estimator interface, it works seamlessly with GridSearchCV. The most impactful hyperparameters to tune are learning_rate, n_estimators (with early stopping), max_depth, subsample, and colsample_bytree. A two-step strategy works well: first set a low learning rate (0.05) and high n_estimators with early stopping to find the right number of trees; then grid search the other parameters with that fixed tree count.
Missing Value Handling in XGBoost
XGBoost natively handles missing values (NaN) without imputation. During tree construction, when a feature has a missing value for some training examples, XGBoost tries both directions (left or right child) for missing values and chooses the direction that maximises the gain. The learned direction is stored in the tree and applied at prediction time. This is a significant advantage over scikit-learn models that require explicit imputation before fitting.
XGBoost Parallel Processing and Speed
Despite trees being built sequentially, XGBoost parallelises the split finding step within each tree: it evaluates all candidate splits across all features simultaneously using multiple CPU threads. Set n_jobs=-1 (or nthread in the native API) to use all available cores. For GPU acceleration, install the CUDA-enabled version and set device='cuda'. On a modern GPU, XGBoost can be 5-50x faster than CPU for large datasets, making it practical for datasets with millions of rows.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
# Use all CPU threads
model_parallel = xgb.XGBRegressor(n_estimators=100, n_jobs=-1, eval_metric='rmse',
verbosity=0, random_state=42)
rmse = np.sqrt(-cross_val_score(model_parallel, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost parallel RMSE:', round(rmse, 4))Quick Check
Test your understanding of XGBoost features from this lesson.
Lesson Recap
In this lesson you learned: XGBoost adds L1/L2 regularisation and second-order gradients to standard gradient boosting, early stopping prevents overfitting by monitoring a validation metric during training, and feature importance can be measured by weight, gain, or cover across all trees. Next up we explore LightGBM's leaf-wise growth strategy and its speed advantages.
Frequently asked questions
Is the “XGBoost: Regularisation, Early Stopping, and Feature Importance” lesson free?
Yes — the full text of “XGBoost: Regularisation, Early Stopping, and Feature Importance” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “XGBoost: Regularisation, Early Stopping, and Feature Importance”?
Learners will train an XGBClassifier, enable early stopping on a validation set, and plot feature importance scores to identify the most predictive columns. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “XGBoost: Regularisation, Early Stopping, and Feature Importance” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Boosting Intuition: Sequential Error Correction
- XGBoost: Regularisation, Early Stopping, and Feature Importance
- LightGBM: Leaf-Wise Growth and Speed Advantages
- Key Hyperparameters: Learning Rate, n_estimators, and max_depth