How KNN Works: Distance, Neighbors, and Votes
Learners will visualise a 2D dataset, compute Euclidean distances, identify the k nearest neighbours, and produce a classification by majority vote.
How KNN Works: Distance, Neighbors, and Votes is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Core Intuition Behind KNN
K-Nearest Neighbors (KNN) is one of the most intuitive machine learning algorithms: predict the label of a new point by looking at the k closest labelled points and taking a majority vote. There is no explicit training phase — the algorithm simply memorises the training data and performs all computation at prediction time. This makes KNN a lazy learner. It works well when similar inputs have similar outputs, which is a reasonable assumption in many real-world problems like recommending products or diagnosing diseases.
# Conceptual pseudocode
def knn_predict(X_train, y_train, x_new, k=3):
# 1. Compute distance from x_new to every training point
distances = [euclidean(x_new, x_i) for x_i in X_train]
# 2. Find indices of k smallest distances
nearest = sorted(range(len(distances)), key=lambda i: distances[i])[:k]
# 3. Majority vote among k neighbors
votes = [y_train[i] for i in nearest]
return max(set(votes), key=votes.count)Euclidean Distance: The Default Metric
The most common distance measure in KNN is Euclidean distance, which is the straight-line distance between two points in feature space. For two points A=(a1, a2) and B=(b1, b2), the Euclidean distance is sqrt((a1-b1)^2 + (a2-b2)^2). In higher dimensions, the same formula extends across all features. Because Euclidean distance treats all dimensions equally, features must be on the same scale — otherwise high-magnitude features dominate the distance calculation and KNN performs poorly.
import numpy as np
def euclidean_distance(a, b):
return np.sqrt(np.sum((a - b) ** 2))
point_a = np.array([1.0, 2.0])
point_b = np.array([4.0, 6.0])
dist = euclidean_distance(point_a, point_b)
print('Euclidean distance:', dist) # 5.0
# Verify with numpy
print('Using numpy:', np.linalg.norm(point_a - point_b))Finding the k Nearest Neighbors
Given a query point, KNN computes distances to all N training points, sorts them, and selects the top k closest. On a small 2D dataset you can visualise this by drawing a circle around the new point that expands until it encloses exactly k training samples — those are the neighbors. The computational cost is O(N * d) per prediction, where N is the number of training points and d is the number of features. This is fine for small datasets but becomes prohibitively slow on large ones.
import numpy as np
# Training data
X_train = np.array([[1,2],[2,3],[3,1],[6,5],[7,7],[8,6]])
y_train = np.array([0, 0, 0, 1, 1, 1]) # 0=class A, 1=class B
# Query point
x_new = np.array([4, 4])
# Distances to all training points
dists = np.linalg.norm(X_train - x_new, axis=1)
print('Distances:', dists.round(2))
# Indices of 3 nearest
k = 3
nearest_idx = np.argsort(dists)[:k]
print('3 nearest labels:', y_train[nearest_idx])Classification by Majority Vote
After finding the k nearest neighbors, KNN for classification assigns the class with the most votes among the neighbors. For k=3, if 2 neighbors are class A and 1 is class B, the prediction is class A. Ties are broken by the implementation (usually by choosing the class with the closest single neighbor). For regression, KNN averages the target values of the k neighbors instead of voting. Choosing k=1 is most flexible but very noisy; larger k is smoother but may underfit.
from collections import Counter
import numpy as np
neighbor_labels = np.array([0, 0, 1]) # 2 votes for class 0, 1 for class 1
# Majority vote
vote_counts = Counter(neighbor_labels)
prediction = vote_counts.most_common(1)[0][0]
print('Predicted class:', prediction) # 0
# For regression: average instead of vote
neighbor_values = np.array([15.2, 18.7, 14.1])
prediction_reg = np.mean(neighbor_values)
print('Regression prediction:', prediction_reg.round(2))KNN with scikit-learn: KNeighborsClassifier
Scikit-learn implements KNN through KNeighborsClassifier and KNeighborsRegressor. These classes follow the standard fit/predict API. Key parameters include n_neighbors (k value), metric (distance function), and weights (uniform or distance-weighted voting). Distance-weighted voting (weights='distance') gives closer neighbors more influence, which often improves performance by reducing the impact of the farthest (least similar) neighbors in the chosen k.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
knn = KNeighborsClassifier(n_neighbors=5, weights='uniform')
knn.fit(X_train_s, y_train)
print('Test accuracy:', knn.score(X_test_s, y_test).round(3))Visualising the Decision Boundary
KNN's decision boundary is inherently non-linear and local. With k=1, the boundary follows training data exactly (creating jagged Voronoi-like regions), while larger k produces smoother boundaries. You can visualise this on a 2D dataset by predicting over a fine mesh grid and colouring each region by predicted class. Low k overfits the training noise (every point is its own class island); high k smooths too aggressively, potentially merging distinct clusters. The ideal k balances this bias-variance trade-off.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsClassifier
# Create 2D mesh for decision boundary
def plot_decision_boundary(clf, X, y):
h = 0.02
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(
np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h)
)
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.4)
plt.scatter(X[:, 0], X[:, 1], c=y)Why Feature Scaling Is Critical for KNN
KNN computes distances in feature space, so the scale of each feature directly affects which points are considered nearest. If one feature has values in the thousands (e.g., income) and another in single digits (e.g., number of children), the high-magnitude feature will dominate all distance calculations. A neighbour that differs by 1 in income but is identical in all other features might be considered farther away than one that differs drastically in all other features. Always apply StandardScaler or MinMaxScaler before KNN.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
# Without scaling
knn_raw = KNeighborsClassifier(n_neighbors=5)
raw_score = cross_val_score(knn_raw, X, y, cv=5).mean()
# With scaling inside pipeline
pipe = Pipeline([('sc', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=5))])
scaled_score = cross_val_score(pipe, X, y, cv=5).mean()
print(f'Without scaling: {raw_score:.3f}')
print(f'With scaling: {scaled_score:.3f}')Getting Neighbor Distances and Indices
Sometimes you need more than just the predicted class — you need to know which training examples were the neighbors and how far away they were. KNN's kneighbors() method returns both the distances and the indices of the nearest training samples. This is useful for anomaly detection (large average distance to neighbors suggests an outlier), recommendation systems, and explaining predictions to end users by showing the most similar known examples.
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
X_train = np.array([[1,2],[2,3],[5,5],[8,7]])
y_train = np.array([0, 0, 1, 1])
knn = KNeighborsClassifier(n_neighbors=2)
knn.fit(X_train, y_train)
x_query = np.array([[4, 4]])
distances, indices = knn.kneighbors(x_query)
print('Neighbor indices:', indices)
print('Distances to neighbors:', distances.round(2))
print('Neighbor labels:', y_train[indices[0]])Weighted Voting by Distance
Uniform voting treats all k neighbors equally regardless of how close they are. Distance-weighted voting (weights='distance') gives each neighbor a weight proportional to the inverse of its distance — very close neighbors have much more influence than far ones. This is particularly useful at decision boundaries where the nearest neighbor and the farthest neighbor may be on different sides of the true boundary. Weighted KNN almost always outperforms uniform KNN, especially when k is large.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
X, y = load_digits(return_X_y=True)
uniform_pipe = Pipeline([('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=7, weights='uniform'))])
distance_pipe = Pipeline([('sc', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=7, weights='distance'))])
print('Uniform: ', cross_val_score(uniform_pipe, X, y, cv=5).mean().round(3))
print('Distance: ', cross_val_score(distance_pipe, X, y, cv=5).mean().round(3))Predicting Class Probabilities
Instead of a hard class label, KNN can return class probabilities using predict_proba(). For k=5, if 3 neighbors are class 1 and 2 are class 0, the predicted probability is [0.4, 0.6]. These probabilities can be thresholded for precision-recall control, used in ensemble models, or calibrated with CalibratedClassifierCV if the raw fractions do not represent true probabilities. For distance-weighted KNN, probabilities are weighted sums rather than simple fractions.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
import numpy as np
X_train = np.array([[1,1],[1,2],[5,5],[6,5],[5,6]])
y_train = np.array([0, 0, 1, 1, 1])
scaler = StandardScaler()
X_s = scaler.fit_transform(X_train)
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_s, y_train)
x_new = scaler.transform([[3, 3]])
proba = knn.predict_proba(x_new)
print('Class probabilities:', proba)
print('Predicted class:', knn.predict(x_new))KNN Strengths and Weaknesses
KNN has several strengths: it is easy to understand, requires no training time, naturally handles multi-class problems, and can model complex non-linear boundaries. Its weaknesses are significant for large datasets: prediction is slow at O(N * d) per query, it requires all training data to be in memory, and it degrades in high dimensions (the curse of dimensionality). KNN is a strong baseline for small to medium datasets with well-scaled features, but is typically replaced by faster models in production at scale.
# Summary of KNN trade-offs
strengths = [
'No training time -- all computation at prediction',
'No assumptions about data distribution',
'Naturally handles multi-class classification',
'Non-linear decision boundary',
]
weaknesses = [
'Slow prediction: O(N*d) per query',
'High memory: stores all training data',
'Sensitive to irrelevant and scaled features',
'Poor in very high dimensions (curse of dimensionality)',
]
for s in strengths: print('+', s)
for w in weaknesses: print('-', w)Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: how KNN classifies by majority vote among k nearest neighbors using Euclidean distance, why feature scaling is critical before applying KNN, and how distance-weighted voting and predict_proba work for more nuanced predictions. Next up we explore how to choose the best k using the elbow method and validation curves.
Frequently asked questions
Is the “How KNN Works: Distance, Neighbors, and Votes” lesson free?
Yes — the full text of “How KNN Works: Distance, Neighbors, and Votes” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “How KNN Works: Distance, Neighbors, and Votes”?
Learners will visualise a 2D dataset, compute Euclidean distances, identify the k nearest neighbours, and produce a classification by majority vote. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “How KNN Works: Distance, Neighbors, and Votes” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- How KNN Works: Distance, Neighbors, and Votes
- Choosing k: The Elbow Method and Validation Curves
- Distance Metrics: Euclidean, Manhattan, and Minkowski
- KNN for Regression and Its Scalability Limits