Machine Learning Academy · 课时

选择 k：肘部法与验证曲线

您将遍历从 1 到 30 的 k 值，绘制验证准确率，并找出平衡偏差与方差的最佳点

选择 k：肘部法与验证曲线 是 CoddyKit 上的免费 Machine Learning Academy 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Choosing k Matters

The value of k is the most important hyperparameter in KNN. Too small a k (e.g., k=1) makes the model highly sensitive to noise: each training point forms its own prediction region, causing the model to memorise noise rather than learn patterns. Too large a k over-smooths the decision boundary and may merge truly distinct classes. Finding the right k is a bias-variance problem: small k = low bias, high variance; large k = high bias, low variance. The elbow method and validation curves help identify the optimal k empirically.

# k=1: memorises training set perfectly
# Training accuracy = 100%, test accuracy low (overfitting)

# k=N (all neighbors): always predicts majority class
# Training accuracy = majority fraction (underfitting)

# Optimal k: somewhere in between
# Maximises test/validation accuracy

from sklearn.neighbors import KNeighborsClassifier
print('k=1  overfits (memorises noise)')
print('k=N  underfits (ignores all variation)')
print('Best k: maximises cross-validated accuracy')

Sweeping k Values: The Basic Loop

The simplest approach is to train KNN for a range of k values, evaluate each on a validation set, and pick the k with the highest validation accuracy. Scikit-learn makes this straightforward: loop over k from 1 to some maximum, fit and score each model. Always evaluate on a held-out validation set or use cross-validation — evaluating on the training set would always select k=1 (since with k=1 KNN predicts training points with 100% accuracy by memorising them).

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_tr = scaler.fit_transform(X_train)
X_v  = scaler.transform(X_val)

val_scores = []
for k in range(1, 31):
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_tr, y_train)
    val_scores.append(knn.score(X_v, y_val))

best_k = val_scores.index(max(val_scores)) + 1
print('Best k:', best_k, 'with accuracy:', max(val_scores).round(3))

Plotting the Validation Curve

Visualising validation accuracy vs k reveals two important patterns. At low k, the curve is noisy with high variance (the model reacts to individual training points). As k increases, accuracy generally improves until a peak, then slowly degrades as the model becomes too smooth. The optimal k is at the peak of the validation curve. This shape is not always obvious with a single validation split, which is why cross-validation gives a more reliable estimate by averaging over multiple splits.

import matplotlib.pyplot as plt
import numpy as np

k_values = range(1, 31)
# val_scores computed from previous sweep
plt.figure(figsize=(10, 5))
plt.plot(k_values, val_scores, marker='o', label='Validation accuracy')
plt.axvline(x=best_k, color='r', linestyle='--', label=f'Best k={best_k}')
plt.xlabel('k (Number of Neighbors)')
plt.ylabel('Validation Accuracy')
plt.title('KNN Validation Curve')
plt.legend()
plt.grid(True)
plt.show()

Cross-Validated Accuracy per k

A single validation split may be biased by which samples happened to land in the validation set. Cross-validation averages across k splits, giving a more stable estimate. Scikit-learn's cross_val_score can be called for each k value. The mean accuracy shows the trend, and the standard deviation across folds shows reliability. Pick the k with the highest mean cross-validated accuracy; if multiple k values are close, prefer the larger one for smoother, more generalisable predictions.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

X, y = load_iris(return_X_y=True)

cv_means, cv_stds = [], []
for k in range(1, 31):
    pipe = Pipeline([('sc', StandardScaler()),
                     ('knn', KNeighborsClassifier(n_neighbors=k))])
    scores = cross_val_score(pipe, X, y, cv=10)
    cv_means.append(scores.mean())
    cv_stds.append(scores.std())

best_k_cv = np.argmax(cv_means) + 1
print('Best k by CV:', best_k_cv, 'mean accuracy:', max(cv_means).round(3))

The Elbow Method Concept

The elbow method is a visual technique for finding the sweet spot where accuracy improvement becomes marginal. Plot validation accuracy (or error) vs k; the curve typically shows steep improvement at small k, then levels off. The elbow — the point where the curve bends from steep to flat — is often the optimal k. The intuition is diminishing returns: adding more neighbors beyond this point does not improve accuracy meaningfully but does increase bias. The elbow is not always obvious, which is why quantitative methods like CV are preferred for final selection.

import matplotlib.pyplot as plt
import numpy as np

k_range = range(1, 31)
error_rates = [1 - acc for acc in cv_means]  # Convert accuracy to error

plt.figure(figsize=(10, 5))
plt.plot(list(k_range), error_rates, marker='o')
plt.xlabel('k')
plt.ylabel('Cross-Validated Error Rate')
plt.title('Elbow Method for Optimal k')
plt.grid(True)

# Mark the elbow visually
plt.axvline(x=best_k_cv, color='r', linestyle='--', label=f'Elbow at k={best_k_cv}')
plt.legend()
plt.show()

Using validation_curve from scikit-learn

Scikit-learn provides validation_curve() as a convenience function that sweeps a hyperparameter range and returns training and validation scores for each value. It is cleaner than a manual loop because it handles cross-validation internally. The param_name argument uses double-underscore notation for pipeline parameters (e.g., knn__n_neighbors). Plotting both train and validation curves together reveals whether low performance is due to underfitting (both low), overfitting (train high, val low), or a good generalisation (both high).

from sklearn.model_selection import validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier())])

train_scores, val_scores = validation_curve(
    pipe, X, y,
    param_name='knn__n_neighbors',
    param_range=range(1, 31),
    cv=10, scoring='accuracy'
)

train_mean = np.mean(train_scores, axis=1)
val_mean   = np.mean(val_scores, axis=1)

print('Best k:', np.argmax(val_mean) + 1)

Bias-Variance Trade-Off in KNN

The validation curve illustrates the bias-variance trade-off directly. For small k: training accuracy approaches 100% (low bias, model fits training data perfectly) while validation accuracy is lower (high variance, model is too sensitive to individual points). For large k: training accuracy drops (model underfits) and validation accuracy also drops (high bias). The optimal k sits at the intersection where the gap between training and validation accuracy is small and both are maximised — this is the generalisation sweet spot.

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(10, 5))
k_range = list(range(1, 31))

ax.plot(k_range, train_mean, label='Training accuracy', color='blue')
ax.plot(k_range, val_mean, label='Validation accuracy', color='orange')
ax.fill_between(k_range,
    np.mean(train_scores, axis=1) - np.std(train_scores, axis=1),
    np.mean(train_scores, axis=1) + np.std(train_scores, axis=1),
    alpha=0.1, color='blue')

ax.set_xlabel('k')
ax.set_ylabel('Accuracy')
ax.set_title('Bias-Variance Trade-off: KNN Validation Curve')
ax.legend()
plt.show()

Choosing k Based on Odd vs Even

For binary classification, always prefer odd values of k to avoid ties. With k=4 and two classes, you might get 2 votes each — the tie-breaking rule then determines the result, which can be arbitrary. By using k=3 or k=5, ties are impossible with two classes. For multi-class problems with C classes, k should not be a multiple of C for the same reason. This is a small but practically important detail when k values near the optimum are close in performance.

# Best practice for binary classification: pick odd k
# For multi-class (C classes): avoid multiples of C

def recommend_k(k_optimal, n_classes):
    if n_classes == 2:
        # Make odd
        return k_optimal if k_optimal % 2 == 1 else k_optimal + 1
    else:
        # Avoid multiples of n_classes
        while k_optimal % n_classes == 0:
            k_optimal += 1
        return k_optimal

print('Binary, k=4 -> recommended:', recommend_k(4, 2))  # 5
print('3-class, k=6 -> recommended:', recommend_k(6, 3)) # 7

GridSearchCV for k Selection

GridSearchCV automates k selection by evaluating every candidate value through cross-validation and returning the best one. Combine it with a scaling pipeline and pass the param_grid with the double-underscore key. GridSearchCV also retrains the best model on the full training set, so grid.best_estimator_ is ready for production use immediately after fitting. This is the recommended approach when k selection is part of a larger hyperparameter optimisation.

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

pipe = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier())
])

param_grid = {
    'knn__n_neighbors': list(range(1, 31, 2)),  # odd values 1-29
    'knn__weights': ['uniform', 'distance']
}

grid = GridSearchCV(pipe, param_grid, cv=10, scoring='accuracy', n_jobs=-1)
grid.fit(X_train, y_train)

print('Best k:', grid.best_params_['knn__n_neighbors'])
print('Best weights:', grid.best_params_['knn__weights'])
print('Best CV accuracy:', grid.best_score_.round(3))

Interpreting Results and Making Final k Choice

When multiple k values produce similar validation scores, prefer the larger k for smoother, more robust predictions that are less sensitive to individual noise points. Inspect the standard deviation of cross-validation scores: if a smaller k has a higher mean but also higher std, the larger k may actually be more reliable in production. The final model should be retrained on the entire training set (not just the cross-validation training folds) using the selected k and evaluated once on the held-out test set.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Select best k from validation
best_k = 11  # determined from CV

# Retrain on full training data
final_model = Pipeline([
    ('sc', StandardScaler()),
    ('knn', KNeighborsClassifier(n_neighbors=best_k, weights='distance'))
])
final_model.fit(X_train, y_train)

# Evaluate once on held-out test set
test_accuracy = final_model.score(X_test, y_test)
print(f'Final test accuracy with k={best_k}: {test_accuracy:.3f}')

Common Pitfalls When Choosing k

Three common mistakes to avoid: (1) Evaluating on training data — k=1 will always score 100%, making it appear optimal; always use held-out or cross-validated data. (2) Not scaling features before choosing k — the optimal k depends on the distance geometry, which changes with scaling; always include the scaler in the pipeline before the hyperparameter search. (3) Choosing k independently of the dataset size — a rule of thumb is to start with k around sqrt(N) where N is training set size, then refine through CV.

import numpy as np

N_train = 1000  # training samples

# Rule of thumb starting point
k_start = int(np.sqrt(N_train))
print(f'sqrt(N) starting point: k = {k_start}')

# Then sweep around this value
k_candidates = list(range(max(1, k_start - 10), k_start + 11, 2))
print('Candidates to sweep:', k_candidates)

# AVOID:
# knn.score(X_train, y_train) -- always pick k=1
# Not including scaler in pipeline before CV

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: how small k causes overfitting and large k causes underfitting, how to use cross-validation and validation_curve to sweep k and find the optimal value, and why odd k values prevent ties in binary classification. Next up we explore different distance metrics — Euclidean, Manhattan, and Minkowski — and how to choose between them.

常见问题解答

「选择 k：肘部法与验证曲线」课时是免费的吗？

是的 — 「选择 k：肘部法与验证曲线」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「选择 k：肘部法与验证曲线」这节课中我会学到什么？

您将遍历从 1 到 30 的 k 值，绘制验证准确率，并找出平衡偏差与方差的最佳点你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「选择 k：肘部法与验证曲线」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Machine Learning Academy