0Pricing
Machine Learning Academy · Lesson

Choosing k: The Elbow Method and Validation Curves

Learners will sweep k from 1 to 30, plot validation accuracy, and identify the sweet spot that balances bias and variance.

Choosing k: The Elbow Method and Validation Curves is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Choosing k Matters

The value of k is the most important hyperparameter in KNN. Too small a k (e.g., k=1) makes the model highly sensitive to noise: each training point forms its own prediction region, causing the model to memorise noise rather than learn patterns. Too large a k over-smooths the decision boundary and may merge truly distinct classes. Finding the right k is a bias-variance problem: small k = low bias, high variance; large k = high bias, low variance. The elbow method and validation curves help identify the optimal k empirically.

# k=1: memorises training set perfectly
# Training accuracy = 100%, test accuracy low (overfitting)

# k=N (all neighbors): always predicts majority class
# Training accuracy = majority fraction (underfitting)

# Optimal k: somewhere in between
# Maximises test/validation accuracy

from sklearn.neighbors import KNeighborsClassifier
print('k=1  overfits (memorises noise)')
print('k=N  underfits (ignores all variation)')
print('Best k: maximises cross-validated accuracy')

Sweeping k Values: The Basic Loop

The simplest approach is to train KNN for a range of k values, evaluate each on a validation set, and pick the k with the highest validation accuracy. Scikit-learn makes this straightforward: loop over k from 1 to some maximum, fit and score each model. Always evaluate on a held-out validation set or use cross-validation — evaluating on the training set would always select k=1 (since with k=1 KNN predicts training points with 100% accuracy by memorising them).

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_tr = scaler.fit_transform(X_train)
X_v  = scaler.transform(X_val)

val_scores = []
for k in range(1, 31):
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_tr, y_train)
    val_scores.append(knn.score(X_v, y_val))

best_k = val_scores.index(max(val_scores)) + 1
print('Best k:', best_k, 'with accuracy:', max(val_scores).round(3))

Plotting the Validation Curve

Visualising validation accuracy vs k reveals two important patterns. At low k, the curve is noisy with high variance (the model reacts to individual training points). As k increases, accuracy generally improves until a peak, then slowly degrades as the model becomes too smooth. The optimal k is at the peak of the validation curve. This shape is not always obvious with a single validation split, which is why cross-validation gives a more reliable estimate by averaging over multiple splits.

import matplotlib.pyplot as plt
import numpy as np

k_values = range(1, 31)
# val_scores computed from previous sweep
plt.figure(figsize=(10, 5))
plt.plot(k_values, val_scores, marker='o', label='Validation accuracy')
plt.axvline(x=best_k, color='r', linestyle='--', label=f'Best k={best_k}')
plt.xlabel('k (Number of Neighbors)')
plt.ylabel('Validation Accuracy')
plt.title('KNN Validation Curve')
plt.legend()
plt.grid(True)
plt.show()

Cross-Validated Accuracy per k

A single validation split may be biased by which samples happened to land in the validation set. Cross-validation averages across k splits, giving a more stable estimate. Scikit-learn's cross_val_score can be called for each k value. The mean accuracy shows the trend, and the standard deviation across folds shows reliability. Pick the k with the highest mean cross-validated accuracy; if multiple k values are close, prefer the larger one for smoother, more generalisable predictions.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

X, y = load_iris(return_X_y=True)

cv_means, cv_stds = [], []
for k in range(1, 31):
    pipe = Pipeline([('sc', StandardScaler()),
                     ('knn', KNeighborsClassifier(n_neighbors=k))])
    scores = cross_val_score(pipe, X, y, cv=10)
    cv_means.append(scores.mean())
    cv_stds.append(scores.std())

best_k_cv = np.argmax(cv_means) + 1
print('Best k by CV:', best_k_cv, 'mean accuracy:', max(cv_means).round(3))

The Elbow Method Concept

The elbow method is a visual technique for finding the sweet spot where accuracy improvement becomes marginal. Plot validation accuracy (or error) vs k; the curve typically shows steep improvement at small k, then levels off. The elbow — the point where the curve bends from steep to flat — is often the optimal k. The intuition is diminishing returns: adding more neighbors beyond this point does not improve accuracy meaningfully but does increase bias. The elbow is not always obvious, which is why quantitative methods like CV are preferred for final selection.

import matplotlib.pyplot as plt
import numpy as np

k_range = range(1, 31)
error_rates = [1 - acc for acc in cv_means]  # Convert accuracy to error

plt.figure(figsize=(10, 5))
plt.plot(list(k_range), error_rates, marker='o')
plt.xlabel('k')
plt.ylabel('Cross-Validated Error Rate')
plt.title('Elbow Method for Optimal k')
plt.grid(True)

# Mark the elbow visually
plt.axvline(x=best_k_cv, color='r', linestyle='--', label=f'Elbow at k={best_k_cv}')
plt.legend()
plt.show()

Using validation_curve from scikit-learn

Scikit-learn provides validation_curve() as a convenience function that sweeps a hyperparameter range and returns training and validation scores for each value. It is cleaner than a manual loop because it handles cross-validation internally. The param_name argument uses double-underscore notation for pipeline parameters (e.g., knn__n_neighbors). Plotting both train and validation curves together reveals whether low performance is due to underfitting (both low), overfitting (train high, val low), or a good generalisation (both high).

from sklearn.model_selection import validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier())])

train_scores, val_scores = validation_curve(
    pipe, X, y,
    param_name='knn__n_neighbors',
    param_range=range(1, 31),
    cv=10, scoring='accuracy'
)

train_mean = np.mean(train_scores, axis=1)
val_mean   = np.mean(val_scores, axis=1)

print('Best k:', np.argmax(val_mean) + 1)

Bias-Variance Trade-Off in KNN

The validation curve illustrates the bias-variance trade-off directly. For small k: training accuracy approaches 100% (low bias, model fits training data perfectly) while validation accuracy is lower (high variance, model is too sensitive to individual points). For large k: training accuracy drops (model underfits) and validation accuracy also drops (high bias). The optimal k sits at the intersection where the gap between training and validation accuracy is small and both are maximised — this is the generalisation sweet spot.

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(10, 5))
k_range = list(range(1, 31))

ax.plot(k_range, train_mean, label='Training accuracy', color='blue')
ax.plot(k_range, val_mean, label='Validation accuracy', color='orange')
ax.fill_between(k_range,
    np.mean(train_scores, axis=1) - np.std(train_scores, axis=1),
    np.mean(train_scores, axis=1) + np.std(train_scores, axis=1),
    alpha=0.1, color='blue')

ax.set_xlabel('k')
ax.set_ylabel('Accuracy')
ax.set_title('Bias-Variance Trade-off: KNN Validation Curve')
ax.legend()
plt.show()

Choosing k Based on Odd vs Even

For binary classification, always prefer odd values of k to avoid ties. With k=4 and two classes, you might get 2 votes each — the tie-breaking rule then determines the result, which can be arbitrary. By using k=3 or k=5, ties are impossible with two classes. For multi-class problems with C classes, k should not be a multiple of C for the same reason. This is a small but practically important detail when k values near the optimum are close in performance.

# Best practice for binary classification: pick odd k
# For multi-class (C classes): avoid multiples of C

def recommend_k(k_optimal, n_classes):
    if n_classes == 2:
        # Make odd
        return k_optimal if k_optimal % 2 == 1 else k_optimal + 1
    else:
        # Avoid multiples of n_classes
        while k_optimal % n_classes == 0:
            k_optimal += 1
        return k_optimal

print('Binary, k=4 -> recommended:', recommend_k(4, 2))  # 5
print('3-class, k=6 -> recommended:', recommend_k(6, 3)) # 7

GridSearchCV for k Selection

GridSearchCV automates k selection by evaluating every candidate value through cross-validation and returning the best one. Combine it with a scaling pipeline and pass the param_grid with the double-underscore key. GridSearchCV also retrains the best model on the full training set, so grid.best_estimator_ is ready for production use immediately after fitting. This is the recommended approach when k selection is part of a larger hyperparameter optimisation.

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

pipe = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier())
])

param_grid = {
    'knn__n_neighbors': list(range(1, 31, 2)),  # odd values 1-29
    'knn__weights': ['uniform', 'distance']
}

grid = GridSearchCV(pipe, param_grid, cv=10, scoring='accuracy', n_jobs=-1)
grid.fit(X_train, y_train)

print('Best k:', grid.best_params_['knn__n_neighbors'])
print('Best weights:', grid.best_params_['knn__weights'])
print('Best CV accuracy:', grid.best_score_.round(3))

Interpreting Results and Making Final k Choice

When multiple k values produce similar validation scores, prefer the larger k for smoother, more robust predictions that are less sensitive to individual noise points. Inspect the standard deviation of cross-validation scores: if a smaller k has a higher mean but also higher std, the larger k may actually be more reliable in production. The final model should be retrained on the entire training set (not just the cross-validation training folds) using the selected k and evaluated once on the held-out test set.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Select best k from validation
best_k = 11  # determined from CV

# Retrain on full training data
final_model = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier(n_neighbors=best_k, weights='distance'))
])
final_model.fit(X_train, y_train)

# Evaluate once on held-out test set
test_accuracy = final_model.score(X_test, y_test)
print(f'Final test accuracy with k={best_k}: {test_accuracy:.3f}')

Common Pitfalls When Choosing k

Three common mistakes to avoid: (1) Evaluating on training data — k=1 will always score 100%, making it appear optimal; always use held-out or cross-validated data. (2) Not scaling features before choosing k — the optimal k depends on the distance geometry, which changes with scaling; always include the scaler in the pipeline before the hyperparameter search. (3) Choosing k independently of the dataset size — a rule of thumb is to start with k around sqrt(N) where N is training set size, then refine through CV.

import numpy as np

N_train = 1000  # training samples

# Rule of thumb starting point
k_start = int(np.sqrt(N_train))
print(f'sqrt(N) starting point: k = {k_start}')

# Then sweep around this value
k_candidates = list(range(max(1, k_start - 10), k_start + 11, 2))
print('Candidates to sweep:', k_candidates)

# AVOID:
# knn.score(X_train, y_train) -- always pick k=1
# Not including scaler in pipeline before CV

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how small k causes overfitting and large k causes underfitting, how to use cross-validation and validation_curve to sweep k and find the optimal value, and why odd k values prevent ties in binary classification. Next up we explore different distance metrics — Euclidean, Manhattan, and Minkowski — and how to choose between them.

Frequently asked questions

Is the “Choosing k: The Elbow Method and Validation Curves” lesson free?

Yes — the full text of “Choosing k: The Elbow Method and Validation Curves” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Choosing k: The Elbow Method and Validation Curves”?

Learners will sweep k from 1 to 30, plot validation accuracy, and identify the sweet spot that balances bias and variance. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Choosing k: The Elbow Method and Validation Curves” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. How KNN Works: Distance, Neighbors, and Votes
  2. Choosing k: The Elbow Method and Validation Curves
  3. Distance Metrics: Euclidean, Manhattan, and Minkowski
  4. KNN for Regression and Its Scalability Limits
← Back to Machine Learning Academy