Tuning C and Gamma with a Grid Search
Learners will run a GridSearchCV over C and gamma, interpret the validation heat map, and select the best hyperparameters for an SVM classifier.
Tuning C and Gamma with a Grid Search is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why C and Gamma Must Be Tuned Together
For an RBF SVM, both C and gamma must be tuned simultaneously because their effects interact. A high C with a small gamma may produce a good boundary, but so might a low C with a high gamma. Searching them independently — fixing one while optimising the other — misses this interaction and often settles on a sub-optimal combination. The standard approach is a 2D grid search that evaluates every combination of C and gamma values and picks the pair with the best cross-validated score.
Setting Up the Grid Search
Use GridSearchCV from scikit-learn to evaluate every hyperparameter combination. Wrap your scaler and SVM in a Pipeline so that GridSearchCV refits the scaler on each training fold without leaking test data statistics. Specify the hyperparameter grid using the double-underscore notation svc__C and svc__gamma to target the named step in the pipeline.
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('scaler', StandardScaler()), ('svc', SVC(kernel='rbf'))])
param_grid = {'svc__C': np.logspace(-2, 3, 6), 'svc__gamma': np.logspace(-4, 0, 5)}
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X, y)
print('Best params:', grid.best_params_)
print('Best CV score:', round(grid.best_score_, 4))Logarithmic Scale for C and Gamma
Both C and gamma should be searched on a logarithmic scale (powers of 10 or powers of 2). The reason is that the model's behaviour changes dramatically as these parameters move from 0.001 to 0.01 (a 10x change), but barely changes from 10 to 11 (a 10% change). A typical coarse grid spans: C in [0.01, 0.1, 1, 10, 100, 1000] and gamma in [0.0001, 0.001, 0.01, 0.1, 1]. After finding the best region, you can refine with a finer grid around those values.
import numpy as np
# Generate logarithmically spaced values
C_range = np.logspace(-2, 3, 6) # [0.01, 0.1, 1, 10, 100, 1000]
gamma_range = np.logspace(-4, 0, 5) # [0.0001, 0.001, 0.01, 0.1, 1.0]
print('C values:', np.round(C_range, 4))
print('gamma values:', np.round(gamma_range, 5))
print('Total combinations:', len(C_range) * len(gamma_range))Interpreting the Validation Heat Map
After a grid search, you can visualise results as a 2D heat map with C on one axis and gamma on the other, coloured by cross-validation score. This reveals: (1) a sweet spot region where scores peak; (2) underfitting at low C and low gamma (too smooth); (3) overfitting at high C and high gamma (too complex). Identifying the shape of the sweet spot helps you understand the model's sensitivity and decide whether to run a finer search.
import numpy as np
import pandas as pd
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC(kernel='rbf'))])
param_grid = {'svc__C': np.logspace(-1, 2, 4), 'svc__gamma': np.logspace(-3, 0, 4)}
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X, y)
results = pd.DataFrame(grid.cv_results_)
pivot = results.pivot_table(index='param_svc__C', columns='param_svc__gamma', values='mean_test_score')
print(pivot.round(3))Coarse-to-Fine Grid Search
A practical strategy is coarse-to-fine search: first run a coarse grid over a wide range to identify the promising region, then run a fine grid focused on that region. For example, if C=10 and gamma=0.01 looks best on the coarse grid, search C in [5, 8, 10, 15, 20] and gamma in [0.005, 0.008, 0.01, 0.015, 0.02] for the fine grid. This reduces total computation compared to running a fine grid over the entire space from the start.
RandomizedSearchCV as an Alternative
For larger grids or when the optimal region is unknown, RandomizedSearchCV samples random combinations from continuous distributions rather than testing every grid point. This scales much better: evaluating 50 random combinations from a 10×10 grid (100 points) finds the optimum nearly as reliably as testing all 100 while using half the compute. Use scipy.stats.loguniform for C and gamma distributions to preserve the logarithmic sampling behaviour.
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
from scipy.stats import loguniform
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC(kernel='rbf'))])
param_dist = {'svc__C': loguniform(0.01, 1000), 'svc__gamma': loguniform(0.0001, 1)}
rnd = RandomizedSearchCV(pipe, param_dist, n_iter=50, cv=5, random_state=42, n_jobs=-1)
rnd.fit(X, y)
print('Best params:', {k: round(v, 5) for k, v in rnd.best_params_.items()})
print('Best CV score:', round(rnd.best_score_, 4))Evaluating the Best Model on the Test Set
After grid search, GridSearchCV.best_estimator_ is automatically refitted on the entire training set with the best hyperparameters. Evaluate it on the held-out test set using grid.score(X_test, y_test) — call this only once, at the very end. If you use the test score to continue tuning, you are effectively leaking test set information into the model selection process and will get an overly optimistic performance estimate.
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC(kernel='rbf'))])
param_grid = {'svc__C': np.logspace(-1, 2, 4), 'svc__gamma': np.logspace(-3, 0, 4)}
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)
print('CV score:', round(grid.best_score_, 4))
print('Test score:', round(grid.score(X_test, y_test), 4))Computation Cost of Grid Search
A grid search with k folds, m C values, and n gamma values trains k × m × n models. A 5-fold search over a 6×5 grid requires 150 model fits. Each fit on a dataset with 1,000 examples and an RBF SVM takes a fraction of a second, but on 100,000 examples it can take minutes per fit. To manage cost: use n_jobs=-1 for parallelism, start with coarse grids, and consider RandomizedSearchCV with fewer iterations for large datasets or expensive models.
Nested Cross-Validation for Unbiased Estimates
Selecting hyperparameters based on validation scores and then evaluating on a separate test set can still give slightly optimistic estimates due to the number of hyperparameter combinations tried. Nested cross-validation uses an outer CV loop for evaluation and an inner CV loop (grid search) for selection. The outer fold test set is never used in any model fitting or hyperparameter selection, giving a truly unbiased generalisation estimate. This is the gold standard for small datasets where you cannot afford a dedicated test set.
Practical Tips for SVM Tuning
Practical advice for SVM grid search: (1) always include your scaler in the pipeline to prevent leakage; (2) use gamma='scale' as the starting point before exploring fixed values; (3) if the best score is at the edge of your grid (e.g., C=1000 is best), extend the grid further; (4) compare the CV score and test score — a large gap indicates overfitting during hyperparameter selection; (5) for multi-class problems, SVM extends automatically using one-vs-one or one-vs-rest strategies under the hood.
SVM Predict_proba via Platt Scaling
By default, SVC does not produce probability estimates — only hard class labels or raw decision function scores. Setting probability=True enables Platt scaling: an additional logistic regression model is fitted on top of the SVM's decision function scores using cross-validation to calibrate the outputs to probabilities. This makes the SVM 5-10x slower to train and slightly changes the decision boundary. Use probability estimates only when you need them (soft voting, ROC-AUC, calibration) and stick to hard predictions otherwise.
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = make_pipeline(StandardScaler(), SVC(kernel='rbf', C=10, gamma=0.01, probability=True))
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)
print('Probability shape:', probs.shape)
print('First 3 predicted probs (class 0, class 1):'); print(probs[:3].round(3))Quick Check
Test your understanding of Grid Search for SVM hyperparameters from this lesson.
Lesson Recap
In this lesson you learned: C and gamma must be tuned together with a 2D grid search, both should be searched on a logarithmic scale to cover relevant magnitudes, and always include the scaler in the pipeline to prevent data leakage during cross-validation. Next up we explore Gradient Boosting and how sequential error correction builds powerful ensembles.
Frequently asked questions
Is the “Tuning C and Gamma with a Grid Search” lesson free?
Yes — the full text of “Tuning C and Gamma with a Grid Search” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Tuning C and Gamma with a Grid Search”?
Learners will run a GridSearchCV over C and gamma, interpret the validation heat map, and select the best hyperparameters for an SVM classifier. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Tuning C and Gamma with a Grid Search” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Maximum Margin Classifier: Support Vectors and Hyperplane
- Soft Margin SVM and the C Parameter
- The Kernel Trick: RBF, Polynomial, and Sigmoid Kernels
- Tuning C and Gamma with a Grid Search