Machine Learning Academy · Lekcja

Grid Search a Random Search

Grid Search a Random Search to bezpłatna lekcja Machine Learning Academy na CoddyKit. To lekcja 3 z 4. Możesz przeczytać całą lekcję poniżej za darmo — a potem ćwiczyć ją interaktywnie w przeglądarce z wbudowanym edytorem kodu i tutorem AI dostępnym 24/7. To część ścieżki edukacyjnej Machine Learning Academy, a Twój postęp synchronizuje się między webem a aplikacją CoddyKit. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.

Części tej lekcji nie zostały jeszcze przetłumaczone i są wyświetlane po angielsku.

The Hyperparameter Search Problem

Most machine learning models have multiple hyperparameters that cannot be learned from data and must be set by the practitioner. Finding the optimal combination manually is impractical — the number of combinations grows exponentially with the number of hyperparameters. Two systematic approaches dominate: Grid Search, which evaluates every combination on a predefined grid, and Random Search, which samples combinations randomly from specified distributions. Understanding when to use each is a crucial practical skill.

Grid Search: Exhaustive Evaluation

GridSearchCV evaluates every combination of the hyperparameter values you specify. For a grid with 4 values of C, 5 values of gamma, and 5 CV folds, it trains 4 × 5 × 5 = 100 models. This guarantees you find the best combination within your grid. However, the cost grows multiplicatively: adding a third hyperparameter with 4 values multiplies the search to 400 models. Grid search works well when you have 1-2 hyperparameters and a manageable grid size.

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_grid = {'svc__C': [0.1, 1, 10], 'svc__gamma': [0.01, 0.1, 1]}
# 3 * 3 * 5 folds = 45 model fits
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X, y)
print('Best params:', grid.best_params_)
print('Best CV score:', round(grid.best_score_, 4))

The Curse of Dimensionality in Grid Search

Grid search suffers from exponential scaling. A model with 5 hyperparameters, each with 5 candidate values, requires 5^5 = 3125 model fits (times the number of CV folds). With 5-fold CV, that is 15,625 training runs. Even if each takes 1 second, that is 4+ hours. Practitioners often restrict grid search to the most important 1-2 hyperparameters, fix others at sensible defaults, and use coarse grids first. This heuristic works but risks missing interactions between hyperparameters not searched jointly.

Random Search: Sampling Instead of Grid

RandomizedSearchCV samples a fixed number of hyperparameter combinations (controlled by n_iter) randomly from specified distributions rather than testing every grid point. Research by Bergstra and Bengio (2012) showed that for the same computational budget, random search finds better hyperparameters than grid search when only a few hyperparameters strongly influence model performance — because random search effectively explores more distinct values of the important parameters.

from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
from scipy.stats import loguniform

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_dist = {'svc__C': loguniform(0.01, 100), 'svc__gamma': loguniform(0.0001, 1)}
# 30 random combinations * 5 folds = 150 model fits (same as a 5*6 grid)
rnd = RandomizedSearchCV(pipe, param_dist, n_iter=30, cv=5, random_state=42, n_jobs=-1)
rnd.fit(X, y)
print('Best params:', {k: round(v, 5) for k, v in rnd.best_params_.items()})
print('Best CV score:', round(rnd.best_score_, 4))

Why Random Search Wins for Many Hyperparameters

Consider 9 hyperparameters, 3 of which matter and 6 of which are irrelevant. A grid with 3 values per parameter tests 3^9 = 19683 combinations but only tests 3 distinct values for each important parameter. With the same budget of 19,683 random samples, each important parameter is explored with 19,683 distinct values. Concentrating samples in the space of important parameters gives random search a decisive advantage. The probability that at least one random configuration is in the top 5% of the space with just 60 iterations is over 95%.

Distributions for Random Search

Choosing appropriate probability distributions for RandomizedSearchCV is important. Use scipy.stats.loguniform(a, b) for parameters that span orders of magnitude like learning_rate or C. Use scipy.stats.uniform(a, b-a) for parameters with a linear scale like subsample (0.5 to 1.0). Use scipy.stats.randint(low, high) for integer parameters like n_estimators or max_depth. Using list-valued parameters (e.g., [3, 5, 7, 9]) samples uniformly from those discrete options.

from scipy.stats import loguniform, uniform, randint
import numpy as np

# Example distributions for RandomForestClassifier + LogisticRegression pipeline
param_dist = {
    'rf__n_estimators': randint(50, 500),          # integer, uniform
    'rf__max_depth': [3, 5, 7, None],              # discrete list
    'rf__min_samples_leaf': randint(1, 20),         # integer, uniform
    'rf__max_features': loguniform(0.1, 1.0)        # continuous, log-scale
}

# Show 5 sample combinations
np.random.seed(42)
for _ in range(3):
    sample = {k: v.rvs() if hasattr(v, 'rvs') else np.random.choice(v) for k, v in param_dist.items()}
    print(sample)

Comparing Grid and Random Search Side by Side

A direct comparison: grid search with 5 values for each of 3 parameters requires 125 fits; random search with 125 iterations covers the same budget but explores continuous distributions instead of 5 fixed points per parameter. In practice, for simple 2D searches (C and gamma for SVM), grid search is perfectly adequate. For complex models like gradient boosting with 6+ hyperparameters, random search with 50-100 iterations consistently outperforms a grid with equivalent computation.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint
import time, numpy as np

X, y = load_breast_cancer(return_X_y=True)
# Grid search
start = time.time()
grid = GridSearchCV(RandomForestClassifier(random_state=42), {'n_estimators': [50,100,200], 'max_depth': [3,5,None]}, cv=3, n_jobs=-1)
grid.fit(X, y)
print(f'Grid search: {round(time.time()-start,1)}s, best={round(grid.best_score_,4)}')
# Random search
start = time.time()
rnd = RandomizedSearchCV(RandomForestClassifier(random_state=42), {'n_estimators': randint(10,300), 'max_depth': [3,5,7,None]}, n_iter=9, cv=3, random_state=42, n_jobs=-1)
rnd.fit(X, y)
print(f'Random search: {round(time.time()-start,1)}s, best={round(rnd.best_score_,4)}')

Halving Grid Search for Larger Spaces

scikit-learn 0.24+ introduced HalvingGridSearchCV and HalvingRandomSearchCV based on the successive halving algorithm: start with all candidates trained on a small data subset, eliminate the worst half, double the data, repeat until one winner remains. This finds good hyperparameters with far less computation than full grid or random search, making it practical for larger search spaces with expensive models.

from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint

X, y = load_breast_cancer(return_X_y=True)
halving = HalvingRandomSearchCV(
    RandomForestClassifier(random_state=42),
    {'n_estimators': randint(10, 500), 'max_depth': [3, 5, 7, None], 'min_samples_leaf': randint(1, 20)},
    cv=3, factor=2, random_state=42, n_jobs=-1
)
halving.fit(X, y)
print('Best params:', halving.best_params_)
print('Best score:', round(halving.best_score_, 4))

Bayesian Optimisation: The Smart Alternative

Both grid and random search are uninformed — they do not use results from previous evaluations to guide future choices. Bayesian optimisation builds a probabilistic model of the objective function (the CV score as a function of hyperparameters) and uses it to intelligently select the most promising combination to evaluate next. Libraries like Optuna, BayesSearchCV (scikit-optimize), and HyperOpt implement this and typically find better hyperparameters in far fewer evaluations than random search.

Practical Search Strategy Recommendations

A practical guideline: (1) start with random search for 30-100 iterations to identify the promising hyperparameter region; (2) if you need more precision, run a fine grid search around that region; (3) for expensive models (slow training), use Bayesian optimisation (Optuna) to minimise evaluations; (4) always use a Pipeline to prevent leakage; (5) set n_jobs=-1 for parallelism; (6) use refit=True (default) so the best model is refitted on the full training data after search. Never use the test set to pick hyperparameters.

Accessing All Results from GridSearchCV

After fitting, grid.cv_results_ is a dictionary containing the mean test score, standard deviation, fit times, and parameter values for every combination evaluated. Converting it to a Pandas DataFrame makes it easy to sort, filter, and visualise. This is useful for understanding the sensitivity of the score to each parameter — if all C values produce similar scores but gamma has a big effect, you know future tuning should focus on gamma.

import pandas as pd
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC(kernel='rbf'))])
grid = GridSearchCV(pipe, {'svc__C': np.logspace(-1, 2, 4), 'svc__gamma': np.logspace(-3, 0, 4)}, cv=5)
grid.fit(X, y)
df = pd.DataFrame(grid.cv_results_)[['param_svc__C', 'param_svc__gamma', 'mean_test_score', 'std_test_score']]
print(df.sort_values('mean_test_score', ascending=False).head(5).round(4))

Quick Check

Test your understanding of Grid Search vs Random Search from this lesson.

Lesson Recap

In this lesson you learned: Grid Search evaluates every combination exhaustively but scales exponentially with hyperparameter count, Random Search samples continuously from distributions and wins when few hyperparameters matter, and Halving and Bayesian optimisation further reduce evaluation cost for expensive models. Next up we explore Nested Cross-Validation for simultaneously selecting and evaluating hyperparameters without bias.

Często zadawane pytania

Czy lekcja „Grid Search a Random Search” jest bezpłatna?

Tak — pełny tekst „Grid Search a Random Search” jest dostępny za darmo tutaj w sieci. Aby ćwiczyć ją interaktywnie (wbudowany edytor kodu i tutor AI dostępny 24/7) i odblokować resztę kursu Machine Learning Academy, przejdź na CoddyKit PRO. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.

Co nauczysz się w „Grid Search a Random Search”?

Uczą się Państwo konfigurować GridSearchCV i RandomizedSearchCV dla tej samej przestrzeni hiperparametrów, porównywać zakres przeszukiwania i koszt obliczeniowy oraz wybierać szybszą metodę dla dużyc… Ćwiczysz Machine Learning Academy z praktycznym kodem, który uruchamiasz bezpośrednio w przeglądarce, a tutor AI dostępny 24/7 odpowiada na Twoje pytania podczas pracy nad lekcją.

Czy potrzebuję doświadczenia, aby zacząć Machine Learning Academy?

Nie wymagamy żadnego doświadczenia. Machine Learning Academy w CoddyKit jest strukturyzowany dla początkujących i zaawansowanych użytkowników, więc możesz zacząć tutaj lub od początku i uczyć się w swoim tempie. To lekcja 3 z 4.

Ile czasu zajmuje lekcja „Grid Search a Random Search”?

Większość lekcji CoddyKit trwa około 5–10 minut. Każda lekcja to mały, interaktywny krok, dzięki czemu robisz systematyczne postępy i zawsze wracasz dokładnie do tego samego miejsca — na webie i w aplikacji.

Czy mogę pisać i uruchamiać kod w tej lekcji Machine Learning Academy?

Tak. Każda lekcja Machine Learning Academy zawiera wbudowany edytor kodu, więc piszesz i uruchamiasz prawdziwy kod bezpośrednio w przeglądarce i od razu otrzymujesz sprzężenie zwrotne od AI — bez konfiguracji na komputerze.

Wszystkie lekcje w tym kursie

← Powrót do Machine Learning Academy