XGBoost: regularización, parada temprana e importancia de las características
Entrene un XGBClassifier, active la parada temprana con un conjunto de validación y represente las puntuaciones de importancia de las características para identificar las columnas más predictivas.
XGBoost: regularización, parada temprana e importancia de las características es una lección gratuita de Machine Learning Academy en CoddyKit. Esta es la lección 2 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de Machine Learning Academy, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de Machine Learning Academy incluye 4 lecciones en total.
Partes de esta lección aún no han sido traducidas y se muestran en inglés.
What Is XGBoost?
XGBoost (eXtreme Gradient Boosting) is a highly optimised gradient boosting library that dominated Kaggle competitions from 2014 onward. It improves on scikit-learn's GradientBoostingClassifier in three major ways: (1) built-in L1 and L2 regularisation on tree weights to reduce overfitting, (2) a second-order Taylor expansion of the loss for more accurate gradient estimates, and (3) a highly efficient approximate histogram-based split finding algorithm that scales to datasets with millions of rows.
Installing and Importing XGBoost
XGBoost is a standalone library installed separately from scikit-learn. It provides a sklearn-compatible API through XGBClassifier and XGBRegressor, so you can use it with cross_val_score, GridSearchCV, and Pipelines just like any scikit-learn estimator. The native XGBoost API uses xgb.DMatrix and xgb.train(), offering more fine-grained control over early stopping and custom objectives.
# Install: pip install xgboost
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1, max_depth=3,
use_label_encoder=False, eval_metric='logloss', random_state=42)
model.fit(X_train, y_train)
print('XGBoost test accuracy:', model.score(X_test, y_test))XGBoost Regularisation: lambda and alpha
XGBoost exposes two regularisation terms: reg_lambda (L2 penalty on leaf weights, default=1) and reg_alpha (L1 penalty on leaf weights, default=0). L2 regularisation shrinks leaf weights toward zero smoothly; L1 can set some leaf weights to exactly zero (sparse tree structure). Both reduce overfitting on noisy datasets. Additionally, min_child_weight requires a minimum sum of instance weights in a child node before a split is made, acting like a minimum-samples-per-leaf constraint.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
for lam in [0, 1, 5, 10]:
model = xgb.XGBClassifier(n_estimators=100, reg_lambda=lam, eval_metric='logloss',
random_state=42, verbosity=0)
score = cross_val_score(model, X, y, cv=5).mean()
print(f'reg_lambda={lam:3d}: CV accuracy={score:.4f}')Early Stopping: Stop When You Stop Improving
Early stopping monitors a validation metric after each boosting round and stops training when the metric has not improved for a specified number of rounds (early_stopping_rounds). This prevents overfitting and saves computation — you can safely set n_estimators very high (e.g., 1000) and let early stopping find the optimal number of rounds. The best iteration is stored in model.best_iteration and is used automatically for predictions.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=3,
eval_metric='logloss', verbosity=0, random_state=42)
model.fit(X_train, y_train,
eval_set=[(X_val, y_val)],
early_stopping_rounds=20,
verbose=False)
print('Best iteration:', model.best_iteration)
print('Test accuracy:', model.score(X_val, y_val))XGBoost Feature Importance
XGBoost provides three types of feature importance: 'weight' (number of times a feature is used in splits), 'gain' (average improvement in loss when a feature is used for splitting — usually the most informative), and 'cover' (average number of samples affected by splits on a feature). Access them via model.feature_importances_ (uses gain by default in the sklearn API) or model.get_booster().get_score(importance_type='gain').
import xgboost as xgb
import pandas as pd
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X, y = data.data, data.target
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(X, y)
importances = pd.Series(model.feature_importances_, index=data.feature_names)
print(importances.sort_values(ascending=False).head(5))Plotting Feature Importance
XGBoost includes a built-in plotting utility xgb.plot_importance(model) that creates a horizontal bar chart of feature importances. For more customisation, use the Series from model.feature_importances_ and plot with matplotlib. Feature importance from boosting is computed differently than from random forests — it reflects how much each feature contributed to reducing the loss across all trees, weighted by usage frequency or average gain.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
import matplotlib.pyplot as plt
data = load_breast_cancer()
model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss', random_state=42)
model.fit(data.data, data.target)
# xgb.plot_importance(model, max_num_features=10) # uncomment in Jupyter
# plt.show()
print('Top feature:', data.feature_names[model.feature_importances_.argmax()])Subsampling Parameters in XGBoost
XGBoost provides three subsampling parameters for additional regularisation: subsample (fraction of training rows used per tree, e.g. 0.8), colsample_bytree (fraction of features used per tree, e.g. 0.8), and colsample_bylevel (fraction of features per depth level). Together these introduce randomness similar to random forests' feature sub-sampling, reducing correlation between trees. Typical starting values: subsample=0.8, colsample_bytree=0.8.
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
X, y = load_breast_cancer(return_X_y=True)
model = xgb.XGBClassifier(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
subsample=0.8,
colsample_bytree=0.8,
reg_lambda=2,
eval_metric='logloss',
random_state=42
)
print('XGBoost with subsampling CV:', cross_val_score(model, X, y, cv=5).mean().round(4))XGBoost for Regression
XGBRegressor uses the same engine but optimises a regression loss (squared error by default, or Tweedie, gamma, quantile, etc.). Early stopping with a regression metric (e.g., RMSE) works exactly the same way. XGBoost is particularly competitive on tabular regression tasks because it handles missing values natively (learns the best direction to send missing-value nodes during tree construction) and supports monotonicity constraints for domain-specific feature relationships.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
model = xgb.XGBRegressor(n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, eval_metric='rmse', random_state=42)
rmse = np.sqrt(-cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost Regression RMSE:', round(rmse, 4))XGBoost with Cross-Validation and GridSearch
Because XGBClassifier implements the scikit-learn estimator interface, it works seamlessly with GridSearchCV. The most impactful hyperparameters to tune are learning_rate, n_estimators (with early stopping), max_depth, subsample, and colsample_bytree. A two-step strategy works well: first set a low learning rate (0.05) and high n_estimators with early stopping to find the right number of trees; then grid search the other parameters with that fixed tree count.
Missing Value Handling in XGBoost
XGBoost natively handles missing values (NaN) without imputation. During tree construction, when a feature has a missing value for some training examples, XGBoost tries both directions (left or right child) for missing values and chooses the direction that maximises the gain. The learned direction is stored in the tree and applied at prediction time. This is a significant advantage over scikit-learn models that require explicit imputation before fitting.
XGBoost Parallel Processing and Speed
Despite trees being built sequentially, XGBoost parallelises the split finding step within each tree: it evaluates all candidate splits across all features simultaneously using multiple CPU threads. Set n_jobs=-1 (or nthread in the native API) to use all available cores. For GPU acceleration, install the CUDA-enabled version and set device='cuda'. On a modern GPU, XGBoost can be 5-50x faster than CPU for large datasets, making it practical for datasets with millions of rows.
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
# Use all CPU threads
model_parallel = xgb.XGBRegressor(n_estimators=100, n_jobs=-1, eval_metric='rmse',
verbosity=0, random_state=42)
rmse = np.sqrt(-cross_val_score(model_parallel, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print('XGBoost parallel RMSE:', round(rmse, 4))Quick Check
Test your understanding of XGBoost features from this lesson.
Lesson Recap
In this lesson you learned: XGBoost adds L1/L2 regularisation and second-order gradients to standard gradient boosting, early stopping prevents overfitting by monitoring a validation metric during training, and feature importance can be measured by weight, gain, or cover across all trees. Next up we explore LightGBM's leaf-wise growth strategy and its speed advantages.
Preguntas frecuentes
¿La lección «XGBoost: regularización, parada temprana e importancia de las características» es gratis?
Sí — el texto completo de «XGBoost: regularización, parada temprana e importancia de las características» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de Machine Learning Academy, actualiza a CoddyKit PRO. El curso de Machine Learning Academy incluye 4 lecciones en total.
¿Qué aprenderé en «XGBoost: regularización, parada temprana e importancia de las características»?
Entrene un XGBClassifier, active la parada temprana con un conjunto de validación y represente las puntuaciones de importancia de las características para identificar las columnas más predictivas. Practicas Machine Learning Academy con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.
¿Necesito experiencia previa para empezar Machine Learning Academy?
No se requiere experiencia previa. Machine Learning Academy en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 2 de 4.
¿Cuánto tiempo toma la lección «XGBoost: regularización, parada temprana e importancia de las características»?
La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.
¿Puedo escribir y ejecutar código en esta lección de Machine Learning Academy?
Sí. Cada lección de Machine Learning Academy incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.
Todas las lecciones de este curso
- Intuición sobre Boosting: corrección secuencial de errores
- XGBoost: regularización, parada temprana e importancia de las características
- LightGBM: crecimiento hoja a hoja y ventajas de velocidad
- Hiperparámetros clave: learning rate, n_estimators y max_depth