Distance Metrics: Euclidean, Manhattan, and Minkowski

KNN defines nearest neighbors using a distance metric — a mathematical function that quantifies how far apart two points are in feature space. The choice of metric directly affects which neighbors are selected, and therefore what the model predicts. Different metrics make different assumptions about the geometry of the data. Euclidean distance assumes diagonal movement is valid; Manhattan distance only allows axis-aligned movement; cosine similarity ignores magnitude and focuses on direction. No single metric is universally best — the right choice depends on the problem structure.

Euclidean distance (also called L2 distance or L2 norm) measures the straight-line distance between two points. In 2D it follows the Pythagorean theorem: sqrt(dx^2 + dy^2). In n dimensions: sqrt(sum of squared differences). It is the most intuitive metric and is the default in KNeighborsClassifier. Euclidean distance works well when features are continuous, on a similar scale, and when the notion of diagonal proximity makes physical sense — for instance, geographic coordinates or sensor readings.

Manhattan distance (L1 norm, city-block distance, taxicab distance) sums the absolute differences along each axis: sum(|a_i - b_i|). The name comes from the grid layout of Manhattan streets — you can only travel along blocks, not diagonally. Manhattan distance is more robust to outliers than Euclidean because it uses absolute values instead of squares. It is often preferred for high-dimensional data and for features that represent counts, ratings, or other quantities where diagonal movement is not physically meaningful.

Minkowski distance is a generalisation that unifies Euclidean and Manhattan under a single formula: (sum(|a_i - b_i|^p))^(1/p). When p=1, it equals Manhattan distance. When p=2, it equals Euclidean distance. When p → infinity, it approaches Chebyshev distance (maximum single-axis difference). In scikit-learn, KNeighborsClassifier uses Minkowski with p=2 by default. You can explore other values of p as a hyperparameter — though p values other than 1 and 2 are rarely used in practice.

Scikit-learn lets you specify the distance metric via the metric parameter. Common string options include 'euclidean', 'manhattan', 'minkowski' (with additional p parameter), 'chebyshev', and 'cosine'. You can also pass a callable Python function as a custom metric. When using non-standard metrics, set algorithm='ball_tree' or algorithm='kd_tree' for efficient neighbor lookup, or algorithm='brute' for a guaranteed-correct but slower exhaustive search.

When comparing Euclidean and Manhattan empirically, the difference appears most clearly in the presence of outlier features. Euclidean squares differences, making one large deviation dominate the total distance. Manhattan sums absolute values, treating all deviations proportionally. In practice, for image data or continuous physical measurements, Euclidean often wins. For high-dimensional sparse data (text, user-item ratings, counts), Manhattan tends to be more stable because it does not amplify the effect of any single dimension.

Cosine similarity measures the angle between two vectors rather than their magnitude. Two documents are considered similar if they point in the same direction in feature space, regardless of document length. Cosine distance = 1 - cosine similarity. This is the preferred metric for text classification with TF-IDF vectors, where two documents can be very different in length but use the same vocabulary in similar proportions. Note that cosine distance is not a true metric (violates triangle inequality) but works well in practice for KNN on text.

Hamming distance counts the number of positions where two vectors differ. It is ideal for binary or categorical features where the concept of magnitude difference is meaningless. For example, comparing two patient records encoded as binary symptom vectors (1=present, 0=absent), Hamming distance counts how many symptoms differ. In scikit-learn, pass metric='hamming' to KNeighborsClassifier. Hamming distance is also used for comparing DNA sequences, error detection in binary codes, and genetic fingerprinting.

As the number of features (dimensions) grows, all distance-based methods suffer from the curse of dimensionality: in high dimensions, the distance between any two random points converges to the same value, making all points appear equally distant. When distances become indistinguishable, the concept of 'nearest neighbor' loses meaning. This is why KNN typically performs best on datasets with fewer than 20-50 features and why dimensionality reduction (PCA, feature selection) is often applied before KNN in high-dimensional settings.

Here is a decision guide for choosing a distance metric: use Euclidean (L2) for continuous features on similar scales (after StandardScaler); use Manhattan (L1) for sparse or high-dimensional data and when outlier robustness is needed; use Cosine for text/TF-IDF vectors where magnitude should not matter; use Hamming for binary or categorical features; use Minkowski with custom p only if you have domain knowledge suggesting a specific geometry. In practice, try Euclidean and Manhattan first using cross-validation and pick the winner.

You can include the metric parameter in your GridSearchCV to find the best combination of k and distance metric simultaneously. This avoids manual trial-and-error across metrics. When searching over metrics that require additional parameters (like Minkowski's p), include those in the param grid too. The best metric is data-dependent and often not obvious from domain knowledge alone — letting cross-validation decide is both principled and practical.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: how Euclidean, Manhattan, and Minkowski distances differ mathematically and when each is appropriate, cosine similarity for text data and Hamming for binary features, and the curse of dimensionality that makes all distances converge in high-dimensional spaces. Next up we explore KNN for regression tasks and its scalability limitations on large datasets.