XGBoost : régularisation, arrêt anticipé et importance des caractéristiques

XGBoost (eXtreme Gradient Boosting) is a highly optimised gradient boosting library that dominated Kaggle competitions from 2014 onward. It improves on scikit-learn's GradientBoostingClassifier in three major ways: (1) built-in L1 and L2 regularisation on tree weights to reduce overfitting, (2) a second-order Taylor expansion of the loss for more accurate gradient estimates, and (3) a highly efficient approximate histogram-based split finding algorithm that scales to datasets with millions of rows.

XGBoost is a standalone library installed separately from scikit-learn. It provides a sklearn-compatible API through XGBClassifier and XGBRegressor, so you can use it with cross_val_score, GridSearchCV, and Pipelines just like any scikit-learn estimator. The native XGBoost API uses xgb.DMatrix and xgb.train(), offering more fine-grained control over early stopping and custom objectives.

XGBoost exposes two regularisation terms: reg_lambda (L2 penalty on leaf weights, default=1) and reg_alpha (L1 penalty on leaf weights, default=0). L2 regularisation shrinks leaf weights toward zero smoothly; L1 can set some leaf weights to exactly zero (sparse tree structure). Both reduce overfitting on noisy datasets. Additionally, min_child_weight requires a minimum sum of instance weights in a child node before a split is made, acting like a minimum-samples-per-leaf constraint.

Early stopping monitors a validation metric after each boosting round and stops training when the metric has not improved for a specified number of rounds (early_stopping_rounds). This prevents overfitting and saves computation — you can safely set n_estimators very high (e.g., 1000) and let early stopping find the optimal number of rounds. The best iteration is stored in model.best_iteration and is used automatically for predictions.

XGBoost provides three types of feature importance: 'weight' (number of times a feature is used in splits), 'gain' (average improvement in loss when a feature is used for splitting — usually the most informative), and 'cover' (average number of samples affected by splits on a feature). Access them via model.feature_importances_ (uses gain by default in the sklearn API) or model.get_booster().get_score(importance_type='gain').

XGBoost includes a built-in plotting utility xgb.plot_importance(model) that creates a horizontal bar chart of feature importances. For more customisation, use the Series from model.feature_importances_ and plot with matplotlib. Feature importance from boosting is computed differently than from random forests — it reflects how much each feature contributed to reducing the loss across all trees, weighted by usage frequency or average gain.

XGBoost provides three subsampling parameters for additional regularisation: subsample (fraction of training rows used per tree, e.g. 0.8), colsample_bytree (fraction of features used per tree, e.g. 0.8), and colsample_bylevel (fraction of features per depth level). Together these introduce randomness similar to random forests' feature sub-sampling, reducing correlation between trees. Typical starting values: subsample=0.8, colsample_bytree=0.8.

XGBRegressor uses the same engine but optimises a regression loss (squared error by default, or Tweedie, gamma, quantile, etc.). Early stopping with a regression metric (e.g., RMSE) works exactly the same way. XGBoost is particularly competitive on tabular regression tasks because it handles missing values natively (learns the best direction to send missing-value nodes during tree construction) and supports monotonicity constraints for domain-specific feature relationships.

Because XGBClassifier implements the scikit-learn estimator interface, it works seamlessly with GridSearchCV. The most impactful hyperparameters to tune are learning_rate, n_estimators (with early stopping), max_depth, subsample, and colsample_bytree. A two-step strategy works well: first set a low learning rate (0.05) and high n_estimators with early stopping to find the right number of trees; then grid search the other parameters with that fixed tree count.

XGBoost natively handles missing values (NaN) without imputation. During tree construction, when a feature has a missing value for some training examples, XGBoost tries both directions (left or right child) for missing values and chooses the direction that maximises the gain. The learned direction is stored in the tree and applied at prediction time. This is a significant advantage over scikit-learn models that require explicit imputation before fitting.

Despite trees being built sequentially, XGBoost parallelises the split finding step within each tree: it evaluates all candidate splits across all features simultaneously using multiple CPU threads. Set n_jobs=-1 (or nthread in the native API) to use all available cores. For GPU acceleration, install the CUDA-enabled version and set device='cuda'. On a modern GPU, XGBoost can be 5-50x faster than CPU for large datasets, making it practical for datasets with millions of rows.

In this lesson you learned: XGBoost adds L1/L2 regularisation and second-order gradients to standard gradient boosting, early stopping prevents overfitting by monitoring a validation metric during training, and feature importance can be measured by weight, gain, or cover across all trees. Next up we explore LightGBM's leaf-wise growth strategy and its speed advantages.