Grid Search vs. Random Search

Most machine learning models have multiple hyperparameters that cannot be learned from data and must be set by the practitioner. Finding the optimal combination manually is impractical — the number of combinations grows exponentially with the number of hyperparameters. Two systematic approaches dominate: Grid Search, which evaluates every combination on a predefined grid, and Random Search, which samples combinations randomly from specified distributions. Understanding when to use each is a crucial practical skill.

GridSearchCV evaluates every combination of the hyperparameter values you specify. For a grid with 4 values of C, 5 values of gamma, and 5 CV folds, it trains 4 × 5 × 5 = 100 models. This guarantees you find the best combination within your grid. However, the cost grows multiplicatively: adding a third hyperparameter with 4 values multiplies the search to 400 models. Grid search works well when you have 1-2 hyperparameters and a manageable grid size.

Grid search suffers from exponential scaling. A model with 5 hyperparameters, each with 5 candidate values, requires 5^5 = 3125 model fits (times the number of CV folds). With 5-fold CV, that is 15,625 training runs. Even if each takes 1 second, that is 4+ hours. Practitioners often restrict grid search to the most important 1-2 hyperparameters, fix others at sensible defaults, and use coarse grids first. This heuristic works but risks missing interactions between hyperparameters not searched jointly.

RandomizedSearchCV samples a fixed number of hyperparameter combinations (controlled by n_iter) randomly from specified distributions rather than testing every grid point. Research by Bergstra and Bengio (2012) showed that for the same computational budget, random search finds better hyperparameters than grid search when only a few hyperparameters strongly influence model performance — because random search effectively explores more distinct values of the important parameters.

Consider 9 hyperparameters, 3 of which matter and 6 of which are irrelevant. A grid with 3 values per parameter tests 3^9 = 19683 combinations but only tests 3 distinct values for each important parameter. With the same budget of 19,683 random samples, each important parameter is explored with 19,683 distinct values. Concentrating samples in the space of important parameters gives random search a decisive advantage. The probability that at least one random configuration is in the top 5% of the space with just 60 iterations is over 95%.

Choosing appropriate probability distributions for RandomizedSearchCV is important. Use scipy.stats.loguniform(a, b) for parameters that span orders of magnitude like learning_rate or C. Use scipy.stats.uniform(a, b-a) for parameters with a linear scale like subsample (0.5 to 1.0). Use scipy.stats.randint(low, high) for integer parameters like n_estimators or max_depth. Using list-valued parameters (e.g., [3, 5, 7, 9]) samples uniformly from those discrete options.

A direct comparison: grid search with 5 values for each of 3 parameters requires 125 fits; random search with 125 iterations covers the same budget but explores continuous distributions instead of 5 fixed points per parameter. In practice, for simple 2D searches (C and gamma for SVM), grid search is perfectly adequate. For complex models like gradient boosting with 6+ hyperparameters, random search with 50-100 iterations consistently outperforms a grid with equivalent computation.

scikit-learn 0.24+ introduced HalvingGridSearchCV and HalvingRandomSearchCV based on the successive halving algorithm: start with all candidates trained on a small data subset, eliminate the worst half, double the data, repeat until one winner remains. This finds good hyperparameters with far less computation than full grid or random search, making it practical for larger search spaces with expensive models.

Both grid and random search are uninformed — they do not use results from previous evaluations to guide future choices. Bayesian optimisation builds a probabilistic model of the objective function (the CV score as a function of hyperparameters) and uses it to intelligently select the most promising combination to evaluate next. Libraries like Optuna, BayesSearchCV (scikit-optimize), and HyperOpt implement this and typically find better hyperparameters in far fewer evaluations than random search.

A practical guideline: (1) start with random search for 30-100 iterations to identify the promising hyperparameter region; (2) if you need more precision, run a fine grid search around that region; (3) for expensive models (slow training), use Bayesian optimisation (Optuna) to minimise evaluations; (4) always use a Pipeline to prevent leakage; (5) set n_jobs=-1 for parallelism; (6) use refit=True (default) so the best model is refitted on the full training data after search. Never use the test set to pick hyperparameters.

After fitting, grid.cv_results_ is a dictionary containing the mean test score, standard deviation, fit times, and parameter values for every combination evaluated. Converting it to a Pandas DataFrame makes it easy to sort, filter, and visualise. This is useful for understanding the sensitivity of the score to each parameter — if all C values produce similar scores but gamma has a big effect, you know future tuning should focus on gamma.

Test your understanding of Grid Search vs Random Search from this lesson.

In this lesson you learned: Grid Search evaluates every combination exhaustively but scales exponentially with hyperparameter count, Random Search samples continuously from distributions and wins when few hyperparameters matter, and Halving and Bayesian optimisation further reduce evaluation cost for expensive models. Next up we explore Nested Cross-Validation for simultaneously selecting and evaluating hyperparameters without bias.