Machine Learning Academy · Lezione

Grid Search vs Random Search

Grid Search vs Random Search è una lezione Machine Learning Academy gratuita su CoddyKit. Questa è la lezione 3 di 4. Puoi leggere la lezione completa qui gratuitamente — poi esercitati direttamente nel browser con un editor di codice integrato e un tutor IA disponibile 24/7. Fa parte del percorso di apprendimento Machine Learning Academy, e i tuoi progressi si sincronizzano tra il web e l'app CoddyKit. Il corso Machine Learning Academy include 4 lezioni in totale.

Parti di questa lezione non sono ancora state tradotte e vengono mostrate in inglese.

The Hyperparameter Search Problem

Most machine learning models have multiple hyperparameters that cannot be learned from data and must be set by the practitioner. Finding the optimal combination manually is impractical — the number of combinations grows exponentially with the number of hyperparameters. Two systematic approaches dominate: Grid Search, which evaluates every combination on a predefined grid, and Random Search, which samples combinations randomly from specified distributions. Understanding when to use each is a crucial practical skill.

Grid Search: Exhaustive Evaluation

GridSearchCV evaluates every combination of the hyperparameter values you specify. For a grid with 4 values of C, 5 values of gamma, and 5 CV folds, it trains 4 × 5 × 5 = 100 models. This guarantees you find the best combination within your grid. However, the cost grows multiplicatively: adding a third hyperparameter with 4 values multiplies the search to 400 models. Grid search works well when you have 1-2 hyperparameters and a manageable grid size.

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_grid = {'svc__C': [0.1, 1, 10], 'svc__gamma': [0.01, 0.1, 1]}
# 3 * 3 * 5 folds = 45 model fits
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X, y)
print('Best params:', grid.best_params_)
print('Best CV score:', round(grid.best_score_, 4))

The Curse of Dimensionality in Grid Search

Grid search suffers from exponential scaling. A model with 5 hyperparameters, each with 5 candidate values, requires 5^5 = 3125 model fits (times the number of CV folds). With 5-fold CV, that is 15,625 training runs. Even if each takes 1 second, that is 4+ hours. Practitioners often restrict grid search to the most important 1-2 hyperparameters, fix others at sensible defaults, and use coarse grids first. This heuristic works but risks missing interactions between hyperparameters not searched jointly.

Random Search: Sampling Instead of Grid

RandomizedSearchCV samples a fixed number of hyperparameter combinations (controlled by n_iter) randomly from specified distributions rather than testing every grid point. Research by Bergstra and Bengio (2012) showed that for the same computational budget, random search finds better hyperparameters than grid search when only a few hyperparameters strongly influence model performance — because random search effectively explores more distinct values of the important parameters.

from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
from scipy.stats import loguniform

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_dist = {'svc__C': loguniform(0.01, 100), 'svc__gamma': loguniform(0.0001, 1)}
# 30 random combinations * 5 folds = 150 model fits (same as a 5*6 grid)
rnd = RandomizedSearchCV(pipe, param_dist, n_iter=30, cv=5, random_state=42, n_jobs=-1)
rnd.fit(X, y)
print('Best params:', {k: round(v, 5) for k, v in rnd.best_params_.items()})
print('Best CV score:', round(rnd.best_score_, 4))

Why Random Search Wins for Many Hyperparameters

Consider 9 hyperparameters, 3 of which matter and 6 of which are irrelevant. A grid with 3 values per parameter tests 3^9 = 19683 combinations but only tests 3 distinct values for each important parameter. With the same budget of 19,683 random samples, each important parameter is explored with 19,683 distinct values. Concentrating samples in the space of important parameters gives random search a decisive advantage. The probability that at least one random configuration is in the top 5% of the space with just 60 iterations is over 95%.

Distributions for Random Search

Choosing appropriate probability distributions for RandomizedSearchCV is important. Use scipy.stats.loguniform(a, b) for parameters that span orders of magnitude like learning_rate or C. Use scipy.stats.uniform(a, b-a) for parameters with a linear scale like subsample (0.5 to 1.0). Use scipy.stats.randint(low, high) for integer parameters like n_estimators or max_depth. Using list-valued parameters (e.g., [3, 5, 7, 9]) samples uniformly from those discrete options.

from scipy.stats import loguniform, uniform, randint
import numpy as np

# Example distributions for RandomForestClassifier + LogisticRegression pipeline
param_dist = {
    'rf__n_estimators': randint(50, 500),          # integer, uniform
    'rf__max_depth': [3, 5, 7, None],              # discrete list
    'rf__min_samples_leaf': randint(1, 20),         # integer, uniform
    'rf__max_features': loguniform(0.1, 1.0)        # continuous, log-scale
}

# Show 5 sample combinations
np.random.seed(42)
for _ in range(3):
    sample = {k: v.rvs() if hasattr(v, 'rvs') else np.random.choice(v) for k, v in param_dist.items()}
    print(sample)

Comparing Grid and Random Search Side by Side

A direct comparison: grid search with 5 values for each of 3 parameters requires 125 fits; random search with 125 iterations covers the same budget but explores continuous distributions instead of 5 fixed points per parameter. In practice, for simple 2D searches (C and gamma for SVM), grid search is perfectly adequate. For complex models like gradient boosting with 6+ hyperparameters, random search with 50-100 iterations consistently outperforms a grid with equivalent computation.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint
import time, numpy as np

X, y = load_breast_cancer(return_X_y=True)
# Grid search
start = time.time()
grid = GridSearchCV(RandomForestClassifier(random_state=42), {'n_estimators': [50,100,200], 'max_depth': [3,5,None]}, cv=3, n_jobs=-1)
grid.fit(X, y)
print(f'Grid search: {round(time.time()-start,1)}s, best={round(grid.best_score_,4)}')
# Random search
start = time.time()
rnd = RandomizedSearchCV(RandomForestClassifier(random_state=42), {'n_estimators': randint(10,300), 'max_depth': [3,5,7,None]}, n_iter=9, cv=3, random_state=42, n_jobs=-1)
rnd.fit(X, y)
print(f'Random search: {round(time.time()-start,1)}s, best={round(rnd.best_score_,4)}')

Halving Grid Search for Larger Spaces

scikit-learn 0.24+ introduced HalvingGridSearchCV and HalvingRandomSearchCV based on the successive halving algorithm: start with all candidates trained on a small data subset, eliminate the worst half, double the data, repeat until one winner remains. This finds good hyperparameters with far less computation than full grid or random search, making it practical for larger search spaces with expensive models.

from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint

X, y = load_breast_cancer(return_X_y=True)
halving = HalvingRandomSearchCV(
    RandomForestClassifier(random_state=42),
    {'n_estimators': randint(10, 500), 'max_depth': [3, 5, 7, None], 'min_samples_leaf': randint(1, 20)},
    cv=3, factor=2, random_state=42, n_jobs=-1
)
halving.fit(X, y)
print('Best params:', halving.best_params_)
print('Best score:', round(halving.best_score_, 4))

Bayesian Optimisation: The Smart Alternative

Both grid and random search are uninformed — they do not use results from previous evaluations to guide future choices. Bayesian optimisation builds a probabilistic model of the objective function (the CV score as a function of hyperparameters) and uses it to intelligently select the most promising combination to evaluate next. Libraries like Optuna, BayesSearchCV (scikit-optimize), and HyperOpt implement this and typically find better hyperparameters in far fewer evaluations than random search.

Practical Search Strategy Recommendations

A practical guideline: (1) start with random search for 30-100 iterations to identify the promising hyperparameter region; (2) if you need more precision, run a fine grid search around that region; (3) for expensive models (slow training), use Bayesian optimisation (Optuna) to minimise evaluations; (4) always use a Pipeline to prevent leakage; (5) set n_jobs=-1 for parallelism; (6) use refit=True (default) so the best model is refitted on the full training data after search. Never use the test set to pick hyperparameters.

Accessing All Results from GridSearchCV

After fitting, grid.cv_results_ is a dictionary containing the mean test score, standard deviation, fit times, and parameter values for every combination evaluated. Converting it to a Pandas DataFrame makes it easy to sort, filter, and visualise. This is useful for understanding the sensitivity of the score to each parameter — if all C values produce similar scores but gamma has a big effect, you know future tuning should focus on gamma.

import pandas as pd
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC(kernel='rbf'))])
grid = GridSearchCV(pipe, {'svc__C': np.logspace(-1, 2, 4), 'svc__gamma': np.logspace(-3, 0, 4)}, cv=5)
grid.fit(X, y)
df = pd.DataFrame(grid.cv_results_)[['param_svc__C', 'param_svc__gamma', 'mean_test_score', 'std_test_score']]
print(df.sort_values('mean_test_score', ascending=False).head(5).round(4))

Quick Check

Test your understanding of Grid Search vs Random Search from this lesson.

Lesson Recap

In this lesson you learned: Grid Search evaluates every combination exhaustively but scales exponentially with hyperparameter count, Random Search samples continuously from distributions and wins when few hyperparameters matter, and Halving and Bayesian optimisation further reduce evaluation cost for expensive models. Next up we explore Nested Cross-Validation for simultaneously selecting and evaluating hyperparameters without bias.

Domande Frequenti

La lezione «Grid Search vs Random Search» è gratuita?

Sì — il testo completo di «Grid Search vs Random Search» è gratuito qui sul web. Per esercitarvi in modo interattivo (un editor di codice integrato e un tutor IA 24/7) e sbloccare il resto del corso Machine Learning Academy, passa a CoddyKit PRO. Il corso Machine Learning Academy include 4 lezioni in totale.

Cosa imparerò in «Grid Search vs Random Search»?

Imparerete a configurare GridSearchCV e RandomizedSearchCV sullo stesso spazio di iperparametri, confrontarne la copertura e il costo computazionale e scegliere il metodo più rapido per gli spazi amp… Eserciti Machine Learning Academy con codice pratico che esegui direttamente nel browser, e un tutor IA 24/7 risponde alle tue domande mentre lavori sulla lezione.

Ho bisogno di esperienza per iniziare Machine Learning Academy?

Non è richiesta alcuna esperienza precedente. Machine Learning Academy su CoddyKit è strutturato per principianti e studenti avanzati, quindi puoi iniziare da qui o dall'inizio e procedere al tuo ritmo. Questa è la lezione 3 di 4.

Quanto tempo richiede la lezione «Grid Search vs Random Search»?

La maggior parte delle lezioni CoddyKit richiede circa 5–10 minuti. Ogni lezione è breve e interattiva, quindi fai progressi costanti e riprendi esattamente da dove hai lasciato su web e app.

Posso scrivere ed eseguire codice in questa lezione Machine Learning Academy?

Sì. Ogni lezione Machine Learning Academy include un editor di codice integrato, quindi scrivi ed esegui codice reale direttamente nel tuo browser e ricevi feedback istantaneo dall'IA — nessuna configurazione locale necessaria.

Tutte le lezioni di questo corso

K-Fold Cross-Validation: suddividere senza data leakage
Cross-validation stratificata e per serie temporali
Grid Search vs Random Search
Cross-validation annidata: selezione e valutazione simultanee

← Torna a Machine Learning Academy