0Pricing
Learn AI with Python · Lesson

Gradient Boosting: GBM and XGBoost

Boosting vs bagging, GradientBoostingClassifier, XGBClassifier, early stopping.

Gradient Boosting: GBM and XGBoost is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Boosting vs Bagging

Bagging trains trees in parallel and averages them. Boosting trains trees sequentially, where each new tree corrects the errors of the previous ones.

Boosting reduces bias, often yielding higher accuracy than bagging on structured data.

Sequential Weak Learners

Gradient boosting builds an ensemble of shallow trees (weak learners). Each tree fits the residuals (gradients of the loss) left by the current ensemble.

Adding many small corrections builds a strong overall model.

GradientBoostingClassifier

Scikit-learn provides GradientBoostingClassifier. Key parameters are n_estimators (number of trees), learning_rate (shrinkage), and max_depth (tree size).

from sklearn.ensemble import GradientBoostingClassifier

gbm = GradientBoostingClassifier(
    n_estimators=200,
    learning_rate=0.1,
    max_depth=3,
    random_state=0,
)
gbm.fit(Xtr, ytr)
print(gbm.score(Xte, yte))

The Learning Rate Tradeoff

learning_rate scales each tree contribution. A smaller learning rate needs more trees but usually generalizes better.

A common strategy: set a low learning rate (e.g. 0.05) and many estimators, then use early stopping.

from sklearn.ensemble import GradientBoostingClassifier

gbm = GradientBoostingClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=3,
)

Why max_depth Stays Small

In boosting, each tree is a weak learner, so max_depth is usually 3 to 6. Deep trees would overfit the residuals quickly. The ensemble gains strength from many shallow trees, not a few deep ones.

Introducing XGBoost

XGBoost is an optimized gradient boosting library: faster, regularized, and with built-in handling for missing values. It is a top performer in tabular ML competitions.

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=300,
    learning_rate=0.1,
    max_depth=4,
    random_state=0,
)
model.fit(Xtr, ytr)
print(model.score(Xte, yte))

Key XGBoost Parameters

The core knobs mirror GBM:

  • n_estimators number of boosting rounds
  • learning_rate shrinkage per round
  • max_depth tree depth
  • subsample and colsample_bytree add randomness for regularization
from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=500,
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    colsample_bytree=0.8,
)

Regularization in XGBoost

XGBoost adds L1 (reg_alpha) and L2 (reg_lambda) penalties plus gamma (minimum loss reduction to split). These control complexity and reduce overfitting beyond what plain GBM offers.

from xgboost import XGBClassifier

model = XGBClassifier(
    reg_alpha=0.1,
    reg_lambda=1.0,
    gamma=0.1,
)

Early Stopping

early_stopping_rounds halts training when the validation metric stops improving for N rounds. This finds the right number of trees automatically and prevents overfitting.

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    early_stopping_rounds=50,
    eval_metric="logloss",
)
model.fit(Xtr, ytr, eval_set=[(Xte, yte)], verbose=False)
print("Best iteration:", model.best_iteration)

Reading Feature Importance

XGBoost exposes feature_importances_ and a richer get_booster().get_score() with importance types like gain and weight. Gain reflects how much a feature improved the loss.

model = XGBClassifier(n_estimators=200).fit(Xtr, ytr)
print(model.feature_importances_)

booster = model.get_booster()
print(booster.get_score(importance_type="gain"))

When to Reach for Boosting

Gradient boosting often beats random forests on tabular data when tuned well. The cost is more sensitivity to hyperparameters and slower sequential training. Start with XGBoost defaults, then tune learning_rate, max_depth, and use early stopping.

Quick Check

Check your boosting knowledge.

Recap

Recap: Boosting trains trees sequentially, each fixing prior errors. Tune n_estimators, learning_rate (low rate + many trees), and small max_depth. XGBoost adds regularization (reg_alpha, reg_lambda, gamma) and early_stopping_rounds to pick the optimal tree count automatically.

Frequently asked questions

Is the “Gradient Boosting: GBM and XGBoost” lesson free?

Yes — the full text of “Gradient Boosting: GBM and XGBoost” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Gradient Boosting: GBM and XGBoost”?

Boosting vs bagging, GradientBoostingClassifier, XGBClassifier, early stopping. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Gradient Boosting: GBM and XGBoost” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Decision Trees: Theory and Implementation
  2. Random Forests and Bagging
  3. Gradient Boosting: GBM and XGBoost
  4. LightGBM and CatBoost
← Back to Learn AI with Python