XGBoost: regolarizzazione, early stopping e importanza delle feature
Addestrerà un XGBClassifier, abiliterà l'early stopping su un set di validazione e traccerà i punteggi di importanza delle feature per identificare le colonne più predittive.
XGBoost: regolarizzazione, early stopping e importanza delle feature è una lezione Machine Learning Academy gratuita su CoddyKit. Questa è la lezione 2 di 4. Puoi leggere la lezione completa qui gratuitamente — poi esercitati direttamente nel browser con un editor di codice integrato e un tutor IA disponibile 24/7. Fa parte del percorso di apprendimento Machine Learning Academy, e i tuoi progressi si sincronizzano tra il web e l'app CoddyKit. Il corso Machine Learning Academy include 4 lezioni in totale.
Parti di questa lezione non sono ancora state tradotte e vengono mostrate in inglese.
What Is XGBoost?
XGBoost (eXtreme Gradient Boosting) is a highly optimised gradient boosting library that dominated Kaggle competitions from 2014 onward. It improves on scikit-learn's GradientBoostingClassifier in three major ways: (1) built-in L1 and L2 regularisation on tree weights to reduce overfitting, (2) a second-order Taylor expansion of the loss for more accurate gradient estimates, and (3) a highly efficient approximate histogram-based split finding algorithm that scales to datasets with millions of rows.
Installing and Importing XGBoost
XGBoost is a standalone library installed separately from scikit-learn. It provides a sklearn-compatible API through XGBClassifier and XGBRegressor, so you can use it with cross_val_score, GridSearchCV, and Pipelines just like any scikit-learn estimator. The native XGBoost API uses xgb.DMatrix and xgb.train(), offering more fine-grained control over early stopping and custom objectives.
# Install: pip install xgboost
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1, max_depth=3,
use_label_encoder=False, eval_metric='logloss', random_state=42)
model.fit(X_train, y_train)
print('XGBoost test accuracy:', model.score(X_test, y_test))XGBoost Regularisation: lambda and alpha
XGBoost exposes two regularisation terms: reg_lambda (L2 penalty on leaf weights, default=1) and reg_alpha (L1 penalty on leaf weights, default=0). L2 regularisation shrinks leaf weights toward zero smoothly; L1 can set some leaf weights to exactly zero (sparse tree structure). Both reduce overfitting on noisy datasets. Additionally, min_child_weight requires a minimum sum of instance weights in a child node before a split is made, acting like a minimum-samples-per-leaf constraint.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
for lam in [0, 1, 5, 10]:
model = xgb.XGBClassifier(n_estimators=100, reg_lambda=lam, eval_metric='logloss',
random_state=42, verbosity=0)
score = cross_val_score(model, X, y, cv=5).mean()
print(f'reg_lambda={lam:3d}: CV accuracy={score:.4f}')Early Stopping: Stop When You Stop Improving
Early stopping monitors a validation metric after each boosting round and stops training when the metric has not improved for a specified number of rounds (early_stopping_rounds). This prevents overfitting and saves computation — you can safely set n_estimators very high (e.g., 1000) and let early stopping find the optimal number of rounds. The best iteration is stored in model.best_iteration and is used automatically for predictions.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=3,
eval_metric='logloss', verbosity=0, random_state=42)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
early_stopping_rounds=20,
verbose=False)
print('Best iteration:', model.best_iteration)
print('Test accuracy:', model.score(X_val, y_val))XGBoost Feature Importance
XGBoost provides three types of feature importance: 'weight' (number of times a feature is used in splits), 'gain' (average improvement in loss when a feature is used for splitting — usually the most informative), and 'cover' (average number of samples affected by splits on a feature). Access them via model.feature_importances_ (uses gain by default in the sklearn API) or model.get_booster().get_score(importance_type='gain').
import xgboost as xgb
import pandas as pd
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X, y = data.data, data.target
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(X, y)
importances = pd.Series(model.feature_importances_, index=data.feature_names)
print(importances.sort_values(ascending=False).head(5))Plotting Feature Importance
XGBoost includes a built-in plotting utility xgb.plot_importance(model) that creates a horizontal bar chart of feature importances. For more customisation, use the Series from model.feature_importances_ and plot with matplotlib. Feature importance from boosting is computed differently than from random forests — it reflects how much each feature contributed to reducing the loss across all trees, weighted by usage frequency or average gain.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
import matplotlib.pyplot as plt
data = load_breast_cancer()
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(data.data, data.target)
# xgb.plot_importance(model, max_num_features=10) # uncomment in Jupyter
# plt.show()
print('Top feature:', data.feature_names[model.feature_importances_.argmax()])Subsampling Parameters in XGBoost
XGBoost provides three subsampling parameters for additional regularisation: subsample (fraction of training rows used per tree, e.g. 0.8), colsample_bytree (fraction of features used per tree, e.g. 0.8), and colsample_bylevel (fraction of features per depth level). Together these introduce randomness similar to random forests' feature sub-sampling, reducing correlation between trees. Typical starting values: subsample=0.8, colsample_bytree=0.8.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
model = xgb.XGBClassifier(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
subsample=0.8,
colsample_bytree=0.8,
reg_lambda=2,
eval_metric='logloss',
random_state=42
)
print('XGBoost with subsampling CV:', cross_val_score(model, X, y, cv=5).mean().round(4))XGBoost for Regression
XGBRegressor uses the same engine but optimises a regression loss (squared error by default, or Tweedie, gamma, quantile, etc.). Early stopping with a regression metric (e.g., RMSE) works exactly the same way. XGBoost is particularly competitive on tabular regression tasks because it handles missing values natively (learns the best direction to send missing-value nodes during tree construction) and supports monotonicity constraints for domain-specific feature relationships.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
model = xgb.XGBRegressor(n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, eval_metric='rmse', random_state=42)
rmse = np.sqrt(-cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost Regression RMSE:', round(rmse, 4))XGBoost with Cross-Validation and GridSearch
Because XGBClassifier implements the scikit-learn estimator interface, it works seamlessly with GridSearchCV. The most impactful hyperparameters to tune are learning_rate, n_estimators (with early stopping), max_depth, subsample, and colsample_bytree. A two-step strategy works well: first set a low learning rate (0.05) and high n_estimators with early stopping to find the right number of trees; then grid search the other parameters with that fixed tree count.
Missing Value Handling in XGBoost
XGBoost natively handles missing values (NaN) without imputation. During tree construction, when a feature has a missing value for some training examples, XGBoost tries both directions (left or right child) for missing values and chooses the direction that maximises the gain. The learned direction is stored in the tree and applied at prediction time. This is a significant advantage over scikit-learn models that require explicit imputation before fitting.
XGBoost Parallel Processing and Speed
Despite trees being built sequentially, XGBoost parallelises the split finding step within each tree: it evaluates all candidate splits across all features simultaneously using multiple CPU threads. Set n_jobs=-1 (or nthread in the native API) to use all available cores. For GPU acceleration, install the CUDA-enabled version and set device='cuda'. On a modern GPU, XGBoost can be 5-50x faster than CPU for large datasets, making it practical for datasets with millions of rows.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
# Use all CPU threads
model_parallel = xgb.XGBRegressor(n_estimators=100, n_jobs=-1, eval_metric='rmse',
verbosity=0, random_state=42)
rmse = np.sqrt(-cross_val_score(model_parallel, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost parallel RMSE:', round(rmse, 4))Quick Check
Test your understanding of XGBoost features from this lesson.
Lesson Recap
In this lesson you learned: XGBoost adds L1/L2 regularisation and second-order gradients to standard gradient boosting, early stopping prevents overfitting by monitoring a validation metric during training, and feature importance can be measured by weight, gain, or cover across all trees. Next up we explore LightGBM's leaf-wise growth strategy and its speed advantages.
Domande Frequenti
La lezione «XGBoost: regolarizzazione, early stopping e importanza delle feature» è gratuita?
Sì — il testo completo di «XGBoost: regolarizzazione, early stopping e importanza delle feature» è gratuito qui sul web. Per esercitarvi in modo interattivo (un editor di codice integrato e un tutor IA 24/7) e sbloccare il resto del corso Machine Learning Academy, passa a CoddyKit PRO. Il corso Machine Learning Academy include 4 lezioni in totale.
Cosa imparerò in «XGBoost: regolarizzazione, early stopping e importanza delle feature»?
Addestrerà un XGBClassifier, abiliterà l'early stopping su un set di validazione e traccerà i punteggi di importanza delle feature per identificare le colonne più predittive. Eserciti Machine Learning Academy con codice pratico che esegui direttamente nel browser, e un tutor IA 24/7 risponde alle tue domande mentre lavori sulla lezione.
Ho bisogno di esperienza per iniziare Machine Learning Academy?
Non è richiesta alcuna esperienza precedente. Machine Learning Academy su CoddyKit è strutturato per principianti e studenti avanzati, quindi puoi iniziare da qui o dall'inizio e procedere al tuo ritmo. Questa è la lezione 2 di 4.
Quanto tempo richiede la lezione «XGBoost: regolarizzazione, early stopping e importanza delle feature»?
La maggior parte delle lezioni CoddyKit richiede circa 5–10 minuti. Ogni lezione è breve e interattiva, quindi fai progressi costanti e riprendi esattamente da dove hai lasciato su web e app.
Posso scrivere ed eseguire codice in questa lezione Machine Learning Academy?
Sì. Ogni lezione Machine Learning Academy include un editor di codice integrato, quindi scrivi ed esegui codice reale direttamente nel tuo browser e ricevi feedback istantaneo dall'IA — nessuna configurazione locale necessaria.
Tutte le lezioni di questo corso
- Intuizione sul boosting: correzione sequenziale degli errori
- XGBoost: regolarizzazione, early stopping e importanza delle feature
- LightGBM: crescita per foglie e vantaggi in termini di velocità
- Iperparametri principali: learning rate, n_estimators e max_depth