XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech
Wytrenuj XGBClassifier, włącz wczesne zatrzymanie na zbiorze walidacyjnym i narysuj wartości ważności cech, aby wskazać najbardziej predykcyjne kolumny.
XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech to bezpłatna lekcja Machine Learning Academy na CoddyKit. To lekcja 2 z 4. Możesz przeczytać całą lekcję poniżej za darmo — a potem ćwiczyć ją interaktywnie w przeglądarce z wbudowanym edytorem kodu i tutorem AI dostępnym 24/7. To część ścieżki edukacyjnej Machine Learning Academy, a Twój postęp synchronizuje się między webem a aplikacją CoddyKit. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.
Części tej lekcji nie zostały jeszcze przetłumaczone i są wyświetlane po angielsku.
What Is XGBoost?
XGBoost (eXtreme Gradient Boosting) is a highly optimised gradient boosting library that dominated Kaggle competitions from 2014 onward. It improves on scikit-learn's GradientBoostingClassifier in three major ways: (1) built-in L1 and L2 regularisation on tree weights to reduce overfitting, (2) a second-order Taylor expansion of the loss for more accurate gradient estimates, and (3) a highly efficient approximate histogram-based split finding algorithm that scales to datasets with millions of rows.
Installing and Importing XGBoost
XGBoost is a standalone library installed separately from scikit-learn. It provides a sklearn-compatible API through XGBClassifier and XGBRegressor, so you can use it with cross_val_score, GridSearchCV, and Pipelines just like any scikit-learn estimator. The native XGBoost API uses xgb.DMatrix and xgb.train(), offering more fine-grained control over early stopping and custom objectives.
# Install: pip install xgboost
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1, max_depth=3,
use_label_encoder=False, eval_metric='logloss', random_state=42)
model.fit(X_train, y_train)
print('XGBoost test accuracy:', model.score(X_test, y_test))XGBoost Regularisation: lambda and alpha
XGBoost exposes two regularisation terms: reg_lambda (L2 penalty on leaf weights, default=1) and reg_alpha (L1 penalty on leaf weights, default=0). L2 regularisation shrinks leaf weights toward zero smoothly; L1 can set some leaf weights to exactly zero (sparse tree structure). Both reduce overfitting on noisy datasets. Additionally, min_child_weight requires a minimum sum of instance weights in a child node before a split is made, acting like a minimum-samples-per-leaf constraint.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
for lam in [0, 1, 5, 10]:
model = xgb.XGBClassifier(n_estimators=100, reg_lambda=lam, eval_metric='logloss',
random_state=42, verbosity=0)
score = cross_val_score(model, X, y, cv=5).mean()
print(f'reg_lambda={lam:3d}: CV accuracy={score:.4f}')Early Stopping: Stop When You Stop Improving
Early stopping monitors a validation metric after each boosting round and stops training when the metric has not improved for a specified number of rounds (early_stopping_rounds). This prevents overfitting and saves computation — you can safely set n_estimators very high (e.g., 1000) and let early stopping find the optimal number of rounds. The best iteration is stored in model.best_iteration and is used automatically for predictions.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=3,
eval_metric='logloss', verbosity=0, random_state=42)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
early_stopping_rounds=20,
verbose=False)
print('Best iteration:', model.best_iteration)
print('Test accuracy:', model.score(X_val, y_val))XGBoost Feature Importance
XGBoost provides three types of feature importance: 'weight' (number of times a feature is used in splits), 'gain' (average improvement in loss when a feature is used for splitting — usually the most informative), and 'cover' (average number of samples affected by splits on a feature). Access them via model.feature_importances_ (uses gain by default in the sklearn API) or model.get_booster().get_score(importance_type='gain').
import xgboost as xgb
import pandas as pd
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X, y = data.data, data.target
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(X, y)
importances = pd.Series(model.feature_importances_, index=data.feature_names)
print(importances.sort_values(ascending=False).head(5))Plotting Feature Importance
XGBoost includes a built-in plotting utility xgb.plot_importance(model) that creates a horizontal bar chart of feature importances. For more customisation, use the Series from model.feature_importances_ and plot with matplotlib. Feature importance from boosting is computed differently than from random forests — it reflects how much each feature contributed to reducing the loss across all trees, weighted by usage frequency or average gain.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
import matplotlib.pyplot as plt
data = load_breast_cancer()
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(data.data, data.target)
# xgb.plot_importance(model, max_num_features=10) # uncomment in Jupyter
# plt.show()
print('Top feature:', data.feature_names[model.feature_importances_.argmax()])Subsampling Parameters in XGBoost
XGBoost provides three subsampling parameters for additional regularisation: subsample (fraction of training rows used per tree, e.g. 0.8), colsample_bytree (fraction of features used per tree, e.g. 0.8), and colsample_bylevel (fraction of features per depth level). Together these introduce randomness similar to random forests' feature sub-sampling, reducing correlation between trees. Typical starting values: subsample=0.8, colsample_bytree=0.8.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
model = xgb.XGBClassifier(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
subsample=0.8,
colsample_bytree=0.8,
reg_lambda=2,
eval_metric='logloss',
random_state=42
)
print('XGBoost with subsampling CV:', cross_val_score(model, X, y, cv=5).mean().round(4))XGBoost for Regression
XGBRegressor uses the same engine but optimises a regression loss (squared error by default, or Tweedie, gamma, quantile, etc.). Early stopping with a regression metric (e.g., RMSE) works exactly the same way. XGBoost is particularly competitive on tabular regression tasks because it handles missing values natively (learns the best direction to send missing-value nodes during tree construction) and supports monotonicity constraints for domain-specific feature relationships.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
model = xgb.XGBRegressor(n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, eval_metric='rmse', random_state=42)
rmse = np.sqrt(-cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost Regression RMSE:', round(rmse, 4))XGBoost with Cross-Validation and GridSearch
Because XGBClassifier implements the scikit-learn estimator interface, it works seamlessly with GridSearchCV. The most impactful hyperparameters to tune are learning_rate, n_estimators (with early stopping), max_depth, subsample, and colsample_bytree. A two-step strategy works well: first set a low learning rate (0.05) and high n_estimators with early stopping to find the right number of trees; then grid search the other parameters with that fixed tree count.
Missing Value Handling in XGBoost
XGBoost natively handles missing values (NaN) without imputation. During tree construction, when a feature has a missing value for some training examples, XGBoost tries both directions (left or right child) for missing values and chooses the direction that maximises the gain. The learned direction is stored in the tree and applied at prediction time. This is a significant advantage over scikit-learn models that require explicit imputation before fitting.
XGBoost Parallel Processing and Speed
Despite trees being built sequentially, XGBoost parallelises the split finding step within each tree: it evaluates all candidate splits across all features simultaneously using multiple CPU threads. Set n_jobs=-1 (or nthread in the native API) to use all available cores. For GPU acceleration, install the CUDA-enabled version and set device='cuda'. On a modern GPU, XGBoost can be 5-50x faster than CPU for large datasets, making it practical for datasets with millions of rows.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
# Use all CPU threads
model_parallel = xgb.XGBRegressor(n_estimators=100, n_jobs=-1, eval_metric='rmse',
verbosity=0, random_state=42)
rmse = np.sqrt(-cross_val_score(model_parallel, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost parallel RMSE:', round(rmse, 4))Quick Check
Test your understanding of XGBoost features from this lesson.
Lesson Recap
In this lesson you learned: XGBoost adds L1/L2 regularisation and second-order gradients to standard gradient boosting, early stopping prevents overfitting by monitoring a validation metric during training, and feature importance can be measured by weight, gain, or cover across all trees. Next up we explore LightGBM's leaf-wise growth strategy and its speed advantages.
Często zadawane pytania
Czy lekcja „XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech” jest bezpłatna?
Tak — pełny tekst „XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech” jest dostępny za darmo tutaj w sieci. Aby ćwiczyć ją interaktywnie (wbudowany edytor kodu i tutor AI dostępny 24/7) i odblokować resztę kursu Machine Learning Academy, przejdź na CoddyKit PRO. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.
Co nauczysz się w „XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech”?
Wytrenuj XGBClassifier, włącz wczesne zatrzymanie na zbiorze walidacyjnym i narysuj wartości ważności cech, aby wskazać najbardziej predykcyjne kolumny. Ćwiczysz Machine Learning Academy z praktycznym kodem, który uruchamiasz bezpośrednio w przeglądarce, a tutor AI dostępny 24/7 odpowiada na Twoje pytania podczas pracy nad lekcją.
Czy potrzebuję doświadczenia, aby zacząć Machine Learning Academy?
Nie wymagamy żadnego doświadczenia. Machine Learning Academy w CoddyKit jest strukturyzowany dla początkujących i zaawansowanych użytkowników, więc możesz zacząć tutaj lub od początku i uczyć się w swoim tempie. To lekcja 2 z 4.
Ile czasu zajmuje lekcja „XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech”?
Większość lekcji CoddyKit trwa około 5–10 minut. Każda lekcja to mały, interaktywny krok, dzięki czemu robisz systematyczne postępy i zawsze wracasz dokładnie do tego samego miejsca — na webie i w aplikacji.
Czy mogę pisać i uruchamiać kod w tej lekcji Machine Learning Academy?
Tak. Każda lekcja Machine Learning Academy zawiera wbudowany edytor kodu, więc piszesz i uruchamiasz prawdziwy kod bezpośrednio w przeglądarce i od razu otrzymujesz sprzężenie zwrotne od AI — bez konfiguracji na komputerze.
Wszystkie lekcje w tym kursie
- Intuicja stojąca za boostingiem: sekwencyjna korekta błędów
- XGBoost: regularyzacja, wczesne zatrzymanie i ważność cech
- LightGBM: rozrost liści i zalety szybkości
- Najważniejsze hiperparametry: learning rate, n_estimators i max_depth