KNN 的工作原理：距离、邻居与投票
您将可视化二维数据集，计算欧氏距离，找出 k 个最近邻，并通过多数投票完成分类
KNN 的工作原理：距离、邻居与投票 是 CoddyKit 上的免费 Machine Learning Academy 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
The Core Intuition Behind KNN
K-Nearest Neighbors (KNN) is one of the most intuitive machine learning algorithms: predict the label of a new point by looking at the k closest labelled points and taking a majority vote. There is no explicit training phase — the algorithm simply memorises the training data and performs all computation at prediction time. This makes KNN a lazy learner. It works well when similar inputs have similar outputs, which is a reasonable assumption in many real-world problems like recommending products or diagnosing diseases.
# Conceptual pseudocode
def knn_predict(X_train, y_train, x_new, k=3):
# 1. Compute distance from x_new to every training point
distances = [euclidean(x_new, x_i) for x_i in X_train]
# 2. Find indices of k smallest distances
nearest = sorted(range(len(distances)), key=lambda i: distances[i])[:k]
# 3. Majority vote among k neighbors
votes = [y_train[i] for i in nearest]
return max(set(votes), key=votes.count)Euclidean Distance: The Default Metric
The most common distance measure in KNN is Euclidean distance, which is the straight-line distance between two points in feature space. For two points A=(a1, a2) and B=(b1, b2), the Euclidean distance is sqrt((a1-b1)^2 + (a2-b2)^2). In higher dimensions, the same formula extends across all features. Because Euclidean distance treats all dimensions equally, features must be on the same scale — otherwise high-magnitude features dominate the distance calculation and KNN performs poorly.
import numpy as np
def euclidean_distance(a, b):
return np.sqrt(np.sum((a - b) ** 2))
point_a = np.array([1.0, 2.0])
point_b = np.array([4.0, 6.0])
dist = euclidean_distance(point_a, point_b)
print('Euclidean distance:', dist) # 5.0
# Verify with numpy
print('Using numpy:', np.linalg.norm(point_a - point_b))Finding the k Nearest Neighbors
Given a query point, KNN computes distances to all N training points, sorts them, and selects the top k closest. On a small 2D dataset you can visualise this by drawing a circle around the new point that expands until it encloses exactly k training samples — those are the neighbors. The computational cost is O(N * d) per prediction, where N is the number of training points and d is the number of features. This is fine for small datasets but becomes prohibitively slow on large ones.
import numpy as np
# Training data
X_train = np.array([[1,2],[2,3],[3,1],[6,5],[7,7],[8,6]])
y_train = np.array([0, 0, 0, 1, 1, 1]) # 0=class A, 1=class B
# Query point
x_new = np.array([4, 4])
# Distances to all training points
dists = np.linalg.norm(X_train - x_new, axis=1)
print('Distances:', dists.round(2))
# Indices of 3 nearest
k = 3
nearest_idx = np.argsort(dists)[:k]
print('3 nearest labels:', y_train[nearest_idx])Classification by Majority Vote
After finding the k nearest neighbors, KNN for classification assigns the class with the most votes among the neighbors. For k=3, if 2 neighbors are class A and 1 is class B, the prediction is class A. Ties are broken by the implementation (usually by choosing the class with the closest single neighbor). For regression, KNN averages the target values of the k neighbors instead of voting. Choosing k=1 is most flexible but very noisy; larger k is smoother but may underfit.
from collections import Counter
import numpy as np
neighbor_labels = np.array([0, 0, 1]) # 2 votes for class 0, 1 for class 1
# Majority vote
vote_counts = Counter(neighbor_labels)
prediction = vote_counts.most_common(1)[0][0]
print('Predicted class:', prediction) # 0
# For regression: average instead of vote
neighbor_values = np.array([15.2, 18.7, 14.1])
prediction_reg = np.mean(neighbor_values)
print('Regression prediction:', prediction_reg.round(2))KNN with scikit-learn: KNeighborsClassifier
Scikit-learn implements KNN through KNeighborsClassifier and KNeighborsRegressor. These classes follow the standard fit/predict API. Key parameters include n_neighbors (k value), metric (distance function), and weights (uniform or distance-weighted voting). Distance-weighted voting (weights='distance') gives closer neighbors more influence, which often improves performance by reducing the impact of the farthest (least similar) neighbors in the chosen k.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
knn = KNeighborsClassifier(n_neighbors=5, weights='uniform')
knn.fit(X_train_s, y_train)
print('Test accuracy:', knn.score(X_test_s, y_test).round(3))Visualising the Decision Boundary
KNN's decision boundary is inherently non-linear and local. With k=1, the boundary follows training data exactly (creating jagged Voronoi-like regions), while larger k produces smoother boundaries. You can visualise this on a 2D dataset by predicting over a fine mesh grid and colouring each region by predicted class. Low k overfits the training noise (every point is its own class island); high k smooths too aggressively, potentially merging distinct clusters. The ideal k balances this bias-variance trade-off.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsClassifier
# Create 2D mesh for decision boundary
def plot_decision_boundary(clf, X, y):
h = 0.02
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(
np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h)
)
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.4)
plt.scatter(X[:, 0], X[:, 1], c=y)Why Feature Scaling Is Critical for KNN
KNN computes distances in feature space, so the scale of each feature directly affects which points are considered nearest. If one feature has values in the thousands (e.g., income) and another in single digits (e.g., number of children), the high-magnitude feature will dominate all distance calculations. A neighbour that differs by 1 in income but is identical in all other features might be considered farther away than one that differs drastically in all other features. Always apply StandardScaler or MinMaxScaler before KNN.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
# Without scaling
knn_raw = KNeighborsClassifier(n_neighbors=5)
raw_score = cross_val_score(knn_raw, X, y, cv=5).mean()
# With scaling inside pipeline
pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=5))])
scaled_score = cross_val_score(pipe, X, y, cv=5).mean()
print(f'Without scaling: {raw_score:.3f}')
print(f'With scaling: {scaled_score:.3f}')Getting Neighbor Distances and Indices
Sometimes you need more than just the predicted class — you need to know which training examples were the neighbors and how far away they were. KNN's kneighbors() method returns both the distances and the indices of the nearest training samples. This is useful for anomaly detection (large average distance to neighbors suggests an outlier), recommendation systems, and explaining predictions to end users by showing the most similar known examples.
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
X_train = np.array([[1,2],[2,3],[5,5],[8,7]])
y_train = np.array([0, 0, 1, 1])
knn = KNeighborsClassifier(n_neighbors=2)
knn.fit(X_train, y_train)
x_query = np.array([[4, 4]])
distances, indices = knn.kneighbors(x_query)
print('Neighbor indices:', indices)
print('Distances to neighbors:', distances.round(2))
print('Neighbor labels:', y_train[indices[0]])Weighted Voting by Distance
Uniform voting treats all k neighbors equally regardless of how close they are. Distance-weighted voting (weights='distance') gives each neighbor a weight proportional to the inverse of its distance — very close neighbors have much more influence than far ones. This is particularly useful at decision boundaries where the nearest neighbor and the farthest neighbor may be on different sides of the true boundary. Weighted KNN almost always outperforms uniform KNN, especially when k is large.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
X, y = load_digits(return_X_y=True)
uniform_pipe = Pipeline([('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=7, weights='uniform'))])
distance_pipe = Pipeline([('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=7, weights='distance'))])
print('Uniform: ', cross_val_score(uniform_pipe, X, y, cv=5).mean().round(3))
print('Distance: ', cross_val_score(distance_pipe, X, y, cv=5).mean().round(3))Predicting Class Probabilities
Instead of a hard class label, KNN can return class probabilities using predict_proba(). For k=5, if 3 neighbors are class 1 and 2 are class 0, the predicted probability is [0.4, 0.6]. These probabilities can be thresholded for precision-recall control, used in ensemble models, or calibrated with CalibratedClassifierCV if the raw fractions do not represent true probabilities. For distance-weighted KNN, probabilities are weighted sums rather than simple fractions.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
import numpy as np
X_train = np.array([[1,1],[1,2],[5,5],[6,5],[5,6]])
y_train = np.array([0, 0, 1, 1, 1])
scaler = StandardScaler()
X_s = scaler.fit_transform(X_train)
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_s, y_train)
x_new = scaler.transform([[3, 3]])
proba = knn.predict_proba(x_new)
print('Class probabilities:', proba)
print('Predicted class:', knn.predict(x_new))KNN Strengths and Weaknesses
KNN has several strengths: it is easy to understand, requires no training time, naturally handles multi-class problems, and can model complex non-linear boundaries. Its weaknesses are significant for large datasets: prediction is slow at O(N * d) per query, it requires all training data to be in memory, and it degrades in high dimensions (the curse of dimensionality). KNN is a strong baseline for small to medium datasets with well-scaled features, but is typically replaced by faster models in production at scale.
# Summary of KNN trade-offs
strengths = [
'No training time -- all computation at prediction',
'No assumptions about data distribution',
'Naturally handles multi-class classification',
'Non-linear decision boundary',
]
weaknesses = [
'Slow prediction: O(N*d) per query',
'High memory: stores all training data',
'Sensitive to irrelevant and scaled features',
'Poor in very high dimensions (curse of dimensionality)',
]
for s in strengths: print('+', s)
for w in weaknesses: print('-', w)Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: how KNN classifies by majority vote among k nearest neighbors using Euclidean distance, why feature scaling is critical before applying KNN, and how distance-weighted voting and predict_proba work for more nuanced predictions. Next up we explore how to choose the best k using the elbow method and validation curves.
常见问题解答
「KNN 的工作原理：距离、邻居与投票」课时是免费的吗？
是的 — 「KNN 的工作原理：距离、邻居与投票」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。
「KNN 的工作原理：距离、邻居与投票」这节课中我会学到什么？
您将可视化二维数据集，计算欧氏距离，找出 k 个最近邻，并通过多数投票完成分类 你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Machine Learning Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。
「KNN 的工作原理：距离、邻居与投票」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Machine Learning Academy 课中编写并运行代码吗？
能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。
此课程中的所有课时
- KNN 的工作原理：距离、邻居与投票
- 选择 k：肘部法与验证曲线
- 距离指标：欧氏距离、曼哈顿距离与闵可夫斯基距离
- 用于回归的 KNN 及其可扩展性限制