Выбор k: метод локтя и кривые валидации
Вы переберёте значения k от 1 до 30, построите график точности на проверочной выборке и определите оптимальное значение, уравновешивающее смещение и дисперсию
«Выбор k: метод локтя и кривые валидации» — бесплатный урок Machine Learning Academy на CoddyKit. Это урок 2 из 4. Ты можешь прочитать весь урок бесплатно ниже — а потом практиковать его прямо в браузере с встроенным редактором кода и ИИ-репетитором 24/7. Это часть пути обучения Machine Learning Academy, и твой прогресс синхронизируется между веб-версией и приложением CoddyKit. Курс Machine Learning Academy содержит 4 уроков всего.
Части этого урока еще не переведены и отображаются на английском.
Why Choosing k Matters
The value of k is the most important hyperparameter in KNN. Too small a k (e.g., k=1) makes the model highly sensitive to noise: each training point forms its own prediction region, causing the model to memorise noise rather than learn patterns. Too large a k over-smooths the decision boundary and may merge truly distinct classes. Finding the right k is a bias-variance problem: small k = low bias, high variance; large k = high bias, low variance. The elbow method and validation curves help identify the optimal k empirically.
# k=1: memorises training set perfectly
# Training accuracy = 100%, test accuracy low (overfitting)
# k=N (all neighbors): always predicts majority class
# Training accuracy = majority fraction (underfitting)
# Optimal k: somewhere in between
# Maximises test/validation accuracy
from sklearn.neighbors import KNeighborsClassifier
print('k=1 overfits (memorises noise)')
print('k=N underfits (ignores all variation)')
print('Best k: maximises cross-validated accuracy')Sweeping k Values: The Basic Loop
The simplest approach is to train KNN for a range of k values, evaluate each on a validation set, and pick the k with the highest validation accuracy. Scikit-learn makes this straightforward: loop over k from 1 to some maximum, fit and score each model. Always evaluate on a held-out validation set or use cross-validation — evaluating on the training set would always select k=1 (since with k=1 KNN predicts training points with 100% accuracy by memorising them).
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_tr = scaler.fit_transform(X_train)
X_v = scaler.transform(X_val)
val_scores = []
for k in range(1, 31):
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(X_tr, y_train)
val_scores.append(knn.score(X_v, y_val))
best_k = val_scores.index(max(val_scores)) + 1
print('Best k:', best_k, 'with accuracy:', max(val_scores).round(3))Plotting the Validation Curve
Visualising validation accuracy vs k reveals two important patterns. At low k, the curve is noisy with high variance (the model reacts to individual training points). As k increases, accuracy generally improves until a peak, then slowly degrades as the model becomes too smooth. The optimal k is at the peak of the validation curve. This shape is not always obvious with a single validation split, which is why cross-validation gives a more reliable estimate by averaging over multiple splits.
import matplotlib.pyplot as plt
import numpy as np
k_values = range(1, 31)
# val_scores computed from previous sweep
plt.figure(figsize=(10, 5))
plt.plot(k_values, val_scores, marker='o', label='Validation accuracy')
plt.axvline(x=best_k, color='r', linestyle='--', label=f'Best k={best_k}')
plt.xlabel('k (Number of Neighbors)')
plt.ylabel('Validation Accuracy')
plt.title('KNN Validation Curve')
plt.legend()
plt.grid(True)
plt.show()Cross-Validated Accuracy per k
A single validation split may be biased by which samples happened to land in the validation set. Cross-validation averages across k splits, giving a more stable estimate. Scikit-learn's cross_val_score can be called for each k value. The mean accuracy shows the trend, and the standard deviation across folds shows reliability. Pick the k with the highest mean cross-validated accuracy; if multiple k values are close, prefer the larger one for smoother, more generalisable predictions.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np
X, y = load_iris(return_X_y=True)
cv_means, cv_stds = [], []
for k in range(1, 31):
pipe = Pipeline([('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=k))])
scores = cross_val_score(pipe, X, y, cv=10)
cv_means.append(scores.mean())
cv_stds.append(scores.std())
best_k_cv = np.argmax(cv_means) + 1
print('Best k by CV:', best_k_cv, 'mean accuracy:', max(cv_means).round(3))The Elbow Method Concept
The elbow method is a visual technique for finding the sweet spot where accuracy improvement becomes marginal. Plot validation accuracy (or error) vs k; the curve typically shows steep improvement at small k, then levels off. The elbow — the point where the curve bends from steep to flat — is often the optimal k. The intuition is diminishing returns: adding more neighbors beyond this point does not improve accuracy meaningfully but does increase bias. The elbow is not always obvious, which is why quantitative methods like CV are preferred for final selection.
import matplotlib.pyplot as plt
import numpy as np
k_range = range(1, 31)
error_rates = [1 - acc for acc in cv_means] # Convert accuracy to error
plt.figure(figsize=(10, 5))
plt.plot(list(k_range), error_rates, marker='o')
plt.xlabel('k')
plt.ylabel('Cross-Validated Error Rate')
plt.title('Elbow Method for Optimal k')
plt.grid(True)
# Mark the elbow visually
plt.axvline(x=best_k_cv, color='r', linestyle='--', label=f'Elbow at k={best_k_cv}')
plt.legend()
plt.show()Using validation_curve from scikit-learn
Scikit-learn provides validation_curve() as a convenience function that sweeps a hyperparameter range and returns training and validation scores for each value. It is cleaner than a manual loop because it handles cross-validation internally. The param_name argument uses double-underscore notation for pipeline parameters (e.g., knn__n_neighbors). Plotting both train and validation curves together reveals whether low performance is due to underfitting (both low), overfitting (train high, val low), or a good generalisation (both high).
from sklearn.model_selection import validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier())])
train_scores, val_scores = validation_curve(
pipe, X, y,
param_name='knn__n_neighbors',
param_range=range(1, 31),
cv=10, scoring='accuracy'
)
train_mean = np.mean(train_scores, axis=1)
val_mean = np.mean(val_scores, axis=1)
print('Best k:', np.argmax(val_mean) + 1)Bias-Variance Trade-Off in KNN
The validation curve illustrates the bias-variance trade-off directly. For small k: training accuracy approaches 100% (low bias, model fits training data perfectly) while validation accuracy is lower (high variance, model is too sensitive to individual points). For large k: training accuracy drops (model underfits) and validation accuracy also drops (high bias). The optimal k sits at the intersection where the gap between training and validation accuracy is small and both are maximised — this is the generalisation sweet spot.
import matplotlib.pyplot as plt
import numpy as np
fig, ax = plt.subplots(figsize=(10, 5))
k_range = list(range(1, 31))
ax.plot(k_range, train_mean, label='Training accuracy', color='blue')
ax.plot(k_range, val_mean, label='Validation accuracy', color='orange')
ax.fill_between(k_range,
np.mean(train_scores, axis=1) - np.std(train_scores, axis=1),
np.mean(train_scores, axis=1) + np.std(train_scores, axis=1),
alpha=0.1, color='blue')
ax.set_xlabel('k')
ax.set_ylabel('Accuracy')
ax.set_title('Bias-Variance Trade-off: KNN Validation Curve')
ax.legend()
plt.show()Choosing k Based on Odd vs Even
For binary classification, always prefer odd values of k to avoid ties. With k=4 and two classes, you might get 2 votes each — the tie-breaking rule then determines the result, which can be arbitrary. By using k=3 or k=5, ties are impossible with two classes. For multi-class problems with C classes, k should not be a multiple of C for the same reason. This is a small but practically important detail when k values near the optimum are close in performance.
# Best practice for binary classification: pick odd k
# For multi-class (C classes): avoid multiples of C
def recommend_k(k_optimal, n_classes):
if n_classes == 2:
# Make odd
return k_optimal if k_optimal % 2 == 1 else k_optimal + 1
else:
# Avoid multiples of n_classes
while k_optimal % n_classes == 0:
k_optimal += 1
return k_optimal
print('Binary, k=4 -> recommended:', recommend_k(4, 2)) # 5
print('3-class, k=6 -> recommended:', recommend_k(6, 3)) # 7GridSearchCV for k Selection
GridSearchCV automates k selection by evaluating every candidate value through cross-validation and returning the best one. Combine it with a scaling pipeline and pass the param_grid with the double-underscore key. GridSearchCV also retrains the best model on the full training set, so grid.best_estimator_ is ready for production use immediately after fitting. This is the recommended approach when k selection is part of a larger hyperparameter optimisation.
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
pipe = Pipeline([
('sc', StandardScaler()),
('knn', KNeighborsClassifier())
])
param_grid = {
'knn__n_neighbors': list(range(1, 31, 2)), # odd values 1-29
'knn__weights': ['uniform', 'distance']
}
grid = GridSearchCV(pipe, param_grid, cv=10, scoring='accuracy', n_jobs=-1)
grid.fit(X_train, y_train)
print('Best k:', grid.best_params_['knn__n_neighbors'])
print('Best weights:', grid.best_params_['knn__weights'])
print('Best CV accuracy:', grid.best_score_.round(3))Interpreting Results and Making Final k Choice
When multiple k values produce similar validation scores, prefer the larger k for smoother, more robust predictions that are less sensitive to individual noise points. Inspect the standard deviation of cross-validation scores: if a smaller k has a higher mean but also higher std, the larger k may actually be more reliable in production. The final model should be retrained on the entire training set (not just the cross-validation training folds) using the selected k and evaluated once on the held-out test set.
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# Select best k from validation
best_k = 11 # determined from CV
# Retrain on full training data
final_model = Pipeline([
('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=best_k, weights='distance'))
])
final_model.fit(X_train, y_train)
# Evaluate once on held-out test set
test_accuracy = final_model.score(X_test, y_test)
print(f'Final test accuracy with k={best_k}: {test_accuracy:.3f}')Common Pitfalls When Choosing k
Three common mistakes to avoid: (1) Evaluating on training data — k=1 will always score 100%, making it appear optimal; always use held-out or cross-validated data. (2) Not scaling features before choosing k — the optimal k depends on the distance geometry, which changes with scaling; always include the scaler in the pipeline before the hyperparameter search. (3) Choosing k independently of the dataset size — a rule of thumb is to start with k around sqrt(N) where N is training set size, then refine through CV.
import numpy as np
N_train = 1000 # training samples
# Rule of thumb starting point
k_start = int(np.sqrt(N_train))
print(f'sqrt(N) starting point: k = {k_start}')
# Then sweep around this value
k_candidates = list(range(max(1, k_start - 10), k_start + 11, 2))
print('Candidates to sweep:', k_candidates)
# AVOID:
# knn.score(X_train, y_train) -- always pick k=1
# Not including scaler in pipeline before CVQuick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: how small k causes overfitting and large k causes underfitting, how to use cross-validation and validation_curve to sweep k and find the optimal value, and why odd k values prevent ties in binary classification. Next up we explore different distance metrics — Euclidean, Manhattan, and Minkowski — and how to choose between them.
Часто задаваемые вопросы
Урок «Выбор k: метод локтя и кривые валидации» бесплатный?
Да — полный текст урока «Выбор k: метод локтя и кривые валидации» бесплатно доступен здесь в веб-версии. Чтобы практиковать его интерактивно (встроенный редактор кода и ИИ-репетитор 24/7) и разблокировать остальной курс Machine Learning Academy, подпишись на CoddyKit PRO. Курс Machine Learning Academy содержит 4 уроков всего.
Чему я научусь в уроке «Выбор k: метод локтя и кривые валидации»?
Вы переберёте значения k от 1 до 30, построите график точности на проверочной выборке и определите оптимальное значение, уравновешивающее смещение и дисперсию Ты практикуешь Machine Learning Academy с помощью реального кода, который запускаешь прямо в браузере, и ИИ-репетитор 24/7 отвечает на твои вопросы во время урока.
Нужен ли мне опыт, чтобы начать Machine Learning Academy?
Предыдущий опыт не требуется. Machine Learning Academy на CoddyKit структурирован для всех уровней — от новичков до продвинутых, поэтому ты можешь начать отсюда или с самого начала и учиться в своем темпе. Это урок 2 из 4.
Сколько времени занимает урок «Выбор k: метод локтя и кривые валидации»?
Большинство уроков CoddyKit занимают около 5–10 минут. Каждый из них компактный и интерактивный, поэтому ты постоянно делаешь прогресс и продолжаешь с того же места в веб-версии и приложении.
Можно ли писать и запускать код в этом уроке Machine Learning Academy?
Да. Каждый урок Machine Learning Academy включает встроенный редактор кода, поэтому ты пишешь и запускаешь реальный код прямо в браузере и получаешь моментальную обратную связь от AI — локальная установка не требуется.
Все уроки этого курса
- Как работает KNN: расстояния, соседи и голоса
- Выбор k: метод локтя и кривые валидации
- Метрики расстояния: евклидова, манхэттенская и Минковского
- KNN для регрессии и ограничения масштабируемости