0Pricing
Machine Learning Academy · Leçon

Choisir k : méthode du coude et courbes de validation

Faites varier k de 1 à 30, tracez la précision de validation et identifiez le meilleur compromis entre biais et variance.

Choisir k : méthode du coude et courbes de validation est une leçon Machine Learning Academy gratuite sur CoddyKit. Ceci est la leçon 2 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Machine Learning Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Machine Learning Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

Why Choosing k Matters

The value of k is the most important hyperparameter in KNN. Too small a k (e.g., k=1) makes the model highly sensitive to noise: each training point forms its own prediction region, causing the model to memorise noise rather than learn patterns. Too large a k over-smooths the decision boundary and may merge truly distinct classes. Finding the right k is a bias-variance problem: small k = low bias, high variance; large k = high bias, low variance. The elbow method and validation curves help identify the optimal k empirically.

# k=1: memorises training set perfectly
# Training accuracy = 100%, test accuracy low (overfitting)

# k=N (all neighbors): always predicts majority class
# Training accuracy = majority fraction (underfitting)

# Optimal k: somewhere in between
# Maximises test/validation accuracy

from sklearn.neighbors import KNeighborsClassifier
print('k=1  overfits (memorises noise)')
print('k=N  underfits (ignores all variation)')
print('Best k: maximises cross-validated accuracy')

Sweeping k Values: The Basic Loop

The simplest approach is to train KNN for a range of k values, evaluate each on a validation set, and pick the k with the highest validation accuracy. Scikit-learn makes this straightforward: loop over k from 1 to some maximum, fit and score each model. Always evaluate on a held-out validation set or use cross-validation — evaluating on the training set would always select k=1 (since with k=1 KNN predicts training points with 100% accuracy by memorising them).

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_tr = scaler.fit_transform(X_train)
X_v  = scaler.transform(X_val)

val_scores = []
for k in range(1, 31):
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_tr, y_train)
    val_scores.append(knn.score(X_v, y_val))

best_k = val_scores.index(max(val_scores)) + 1
print('Best k:', best_k, 'with accuracy:', max(val_scores).round(3))

Plotting the Validation Curve

Visualising validation accuracy vs k reveals two important patterns. At low k, the curve is noisy with high variance (the model reacts to individual training points). As k increases, accuracy generally improves until a peak, then slowly degrades as the model becomes too smooth. The optimal k is at the peak of the validation curve. This shape is not always obvious with a single validation split, which is why cross-validation gives a more reliable estimate by averaging over multiple splits.

import matplotlib.pyplot as plt
import numpy as np

k_values = range(1, 31)
# val_scores computed from previous sweep
plt.figure(figsize=(10, 5))
plt.plot(k_values, val_scores, marker='o', label='Validation accuracy')
plt.axvline(x=best_k, color='r', linestyle='--', label=f'Best k={best_k}')
plt.xlabel('k (Number of Neighbors)')
plt.ylabel('Validation Accuracy')
plt.title('KNN Validation Curve')
plt.legend()
plt.grid(True)
plt.show()

Cross-Validated Accuracy per k

A single validation split may be biased by which samples happened to land in the validation set. Cross-validation averages across k splits, giving a more stable estimate. Scikit-learn's cross_val_score can be called for each k value. The mean accuracy shows the trend, and the standard deviation across folds shows reliability. Pick the k with the highest mean cross-validated accuracy; if multiple k values are close, prefer the larger one for smoother, more generalisable predictions.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

X, y = load_iris(return_X_y=True)

cv_means, cv_stds = [], []
for k in range(1, 31):
    pipe = Pipeline([('sc', StandardScaler()),
                     ('knn', KNeighborsClassifier(n_neighbors=k))])
    scores = cross_val_score(pipe, X, y, cv=10)
    cv_means.append(scores.mean())
    cv_stds.append(scores.std())

best_k_cv = np.argmax(cv_means) + 1
print('Best k by CV:', best_k_cv, 'mean accuracy:', max(cv_means).round(3))

The Elbow Method Concept

The elbow method is a visual technique for finding the sweet spot where accuracy improvement becomes marginal. Plot validation accuracy (or error) vs k; the curve typically shows steep improvement at small k, then levels off. The elbow — the point where the curve bends from steep to flat — is often the optimal k. The intuition is diminishing returns: adding more neighbors beyond this point does not improve accuracy meaningfully but does increase bias. The elbow is not always obvious, which is why quantitative methods like CV are preferred for final selection.

import matplotlib.pyplot as plt
import numpy as np

k_range = range(1, 31)
error_rates = [1 - acc for acc in cv_means]  # Convert accuracy to error

plt.figure(figsize=(10, 5))
plt.plot(list(k_range), error_rates, marker='o')
plt.xlabel('k')
plt.ylabel('Cross-Validated Error Rate')
plt.title('Elbow Method for Optimal k')
plt.grid(True)

# Mark the elbow visually
plt.axvline(x=best_k_cv, color='r', linestyle='--', label=f'Elbow at k={best_k_cv}')
plt.legend()
plt.show()

Using validation_curve from scikit-learn

Scikit-learn provides validation_curve() as a convenience function that sweeps a hyperparameter range and returns training and validation scores for each value. It is cleaner than a manual loop because it handles cross-validation internally. The param_name argument uses double-underscore notation for pipeline parameters (e.g., knn__n_neighbors). Plotting both train and validation curves together reveals whether low performance is due to underfitting (both low), overfitting (train high, val low), or a good generalisation (both high).

from sklearn.model_selection import validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier())])

train_scores, val_scores = validation_curve(
    pipe, X, y,
    param_name='knn__n_neighbors',
    param_range=range(1, 31),
    cv=10, scoring='accuracy'
)

train_mean = np.mean(train_scores, axis=1)
val_mean   = np.mean(val_scores, axis=1)

print('Best k:', np.argmax(val_mean) + 1)

Bias-Variance Trade-Off in KNN

The validation curve illustrates the bias-variance trade-off directly. For small k: training accuracy approaches 100% (low bias, model fits training data perfectly) while validation accuracy is lower (high variance, model is too sensitive to individual points). For large k: training accuracy drops (model underfits) and validation accuracy also drops (high bias). The optimal k sits at the intersection where the gap between training and validation accuracy is small and both are maximised — this is the generalisation sweet spot.

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(10, 5))
k_range = list(range(1, 31))

ax.plot(k_range, train_mean, label='Training accuracy', color='blue')
ax.plot(k_range, val_mean, label='Validation accuracy', color='orange')
ax.fill_between(k_range,
    np.mean(train_scores, axis=1) - np.std(train_scores, axis=1),
    np.mean(train_scores, axis=1) + np.std(train_scores, axis=1),
    alpha=0.1, color='blue')

ax.set_xlabel('k')
ax.set_ylabel('Accuracy')
ax.set_title('Bias-Variance Trade-off: KNN Validation Curve')
ax.legend()
plt.show()

Choosing k Based on Odd vs Even

For binary classification, always prefer odd values of k to avoid ties. With k=4 and two classes, you might get 2 votes each — the tie-breaking rule then determines the result, which can be arbitrary. By using k=3 or k=5, ties are impossible with two classes. For multi-class problems with C classes, k should not be a multiple of C for the same reason. This is a small but practically important detail when k values near the optimum are close in performance.

# Best practice for binary classification: pick odd k
# For multi-class (C classes): avoid multiples of C

def recommend_k(k_optimal, n_classes):
    if n_classes == 2:
        # Make odd
        return k_optimal if k_optimal % 2 == 1 else k_optimal + 1
    else:
        # Avoid multiples of n_classes
        while k_optimal % n_classes == 0:
            k_optimal += 1
        return k_optimal

print('Binary, k=4 -> recommended:', recommend_k(4, 2))  # 5
print('3-class, k=6 -> recommended:', recommend_k(6, 3)) # 7

GridSearchCV for k Selection

GridSearchCV automates k selection by evaluating every candidate value through cross-validation and returning the best one. Combine it with a scaling pipeline and pass the param_grid with the double-underscore key. GridSearchCV also retrains the best model on the full training set, so grid.best_estimator_ is ready for production use immediately after fitting. This is the recommended approach when k selection is part of a larger hyperparameter optimisation.

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

pipe = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier())
])

param_grid = {
    'knn__n_neighbors': list(range(1, 31, 2)),  # odd values 1-29
    'knn__weights': ['uniform', 'distance']
}

grid = GridSearchCV(pipe, param_grid, cv=10, scoring='accuracy', n_jobs=-1)
grid.fit(X_train, y_train)

print('Best k:', grid.best_params_['knn__n_neighbors'])
print('Best weights:', grid.best_params_['knn__weights'])
print('Best CV accuracy:', grid.best_score_.round(3))

Interpreting Results and Making Final k Choice

When multiple k values produce similar validation scores, prefer the larger k for smoother, more robust predictions that are less sensitive to individual noise points. Inspect the standard deviation of cross-validation scores: if a smaller k has a higher mean but also higher std, the larger k may actually be more reliable in production. The final model should be retrained on the entire training set (not just the cross-validation training folds) using the selected k and evaluated once on the held-out test set.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Select best k from validation
best_k = 11  # determined from CV

# Retrain on full training data
final_model = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier(n_neighbors=best_k, weights='distance'))
])
final_model.fit(X_train, y_train)

# Evaluate once on held-out test set
test_accuracy = final_model.score(X_test, y_test)
print(f'Final test accuracy with k={best_k}: {test_accuracy:.3f}')

Common Pitfalls When Choosing k

Three common mistakes to avoid: (1) Evaluating on training data — k=1 will always score 100%, making it appear optimal; always use held-out or cross-validated data. (2) Not scaling features before choosing k — the optimal k depends on the distance geometry, which changes with scaling; always include the scaler in the pipeline before the hyperparameter search. (3) Choosing k independently of the dataset size — a rule of thumb is to start with k around sqrt(N) where N is training set size, then refine through CV.

import numpy as np

N_train = 1000  # training samples

# Rule of thumb starting point
k_start = int(np.sqrt(N_train))
print(f'sqrt(N) starting point: k = {k_start}')

# Then sweep around this value
k_candidates = list(range(max(1, k_start - 10), k_start + 11, 2))
print('Candidates to sweep:', k_candidates)

# AVOID:
# knn.score(X_train, y_train) -- always pick k=1
# Not including scaler in pipeline before CV

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how small k causes overfitting and large k causes underfitting, how to use cross-validation and validation_curve to sweep k and find the optimal value, and why odd k values prevent ties in binary classification. Next up we explore different distance metrics — Euclidean, Manhattan, and Minkowski — and how to choose between them.

Questions Fréquemment Posées

La leçon « Choisir k : méthode du coude et courbes de validation » est-elle gratuite ?

Oui — le texte complet de « Choisir k : méthode du coude et courbes de validation » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Machine Learning Academy, passe à CoddyKit PRO. Le cours Machine Learning Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Choisir k : méthode du coude et courbes de validation » ?

Faites varier k de 1 à 30, tracez la précision de validation et identifiez le meilleur compromis entre biais et variance. Tu pratiques Machine Learning Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer Machine Learning Academy ?

Aucune expérience préalable n'est requise. Machine Learning Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 2 sur 4.

Combien de temps prend la leçon « Choisir k : méthode du coude et courbes de validation » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon Machine Learning Academy ?

Oui. Chaque leçon Machine Learning Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

  1. Fonctionnement de KNN : distance, voisins et votes
  2. Choisir k : méthode du coude et courbes de validation
  3. Métriques de distance : euclidienne, de Manhattan et de Minkowski
  4. KNN pour la régression et limites de passage à l’échelle
← Retour à Machine Learning Academy