K wählen: Elbow-Methode und Silhouettenkoeffizient

K-Means requires you to specify k — the number of clusters — before training. Too few clusters and you lump distinct groups together; too many and you split natural groups artificially. There is no universally correct k, but two diagnostic tools — the elbow method and the silhouette score — give principled guidance.

As you increase k, inertia always decreases because points are assigned to closer centroids. At k=n (one cluster per point), inertia is zero. This means you cannot simply minimise inertia — you need to find where additional clusters stop providing meaningful reductions. That point of diminishing returns is the elbow.

Plot inertia on the y-axis against k on the x-axis. The curve typically drops steeply for the first few k values then flattens. The elbow — the kink where the rate of decrease sharply slows — is your estimate of the true cluster count. If the true k is 3, the drop from k=1 to k=3 is large, but from k=3 to k=4 is much smaller.

The elbow method works well when clusters are clearly separated, but real-world data often produces a smooth curve with no obvious kink. In such cases the elbow is ambiguous and different people may pick different k. That is where the silhouette score provides a more objective, mathematically grounded alternative.

For each point i, compute two values: a(i) = mean distance to other points in the same cluster (cohesion), and b(i) = mean distance to the nearest different cluster (separation). The silhouette for point i is s(i) = (b(i) - a(i)) / max(a(i), b(i)). Values range from −1 (wrong cluster) through 0 (on border) to +1 (tight, well-separated cluster).

sklearn.metrics.silhouette_score returns the mean silhouette over all points. A score above 0.5 typically indicates reasonable clustering; above 0.7 is strong. Because you cannot compute the silhouette for k=1 (no second cluster), sweep k from 2 to some maximum and pick the k with the highest mean score.

A silhouette plot shows the silhouette coefficient of every individual point, sorted by cluster and width. Wide, uniform bars indicate all points are well-placed. Thin bars or points with negative scores reveal misassigned outliers. scikit-learn's silhouette_samples returns per-point scores that you can visualise this way.

In practice, use both methods together. If the elbow suggests k=4 and the silhouette score is also highest at k=4, you have strong convergent evidence. When they disagree — e.g., elbow at k=3 but silhouette peaks at k=5 — examine the silhouette plot for each candidate k and apply domain knowledge to make the final call.

The gap statistic compares the observed inertia against the expected inertia under a null reference distribution (data sampled uniformly in the feature space). Choose the smallest k where gap(k) >= gap(k+1) - stddev. It is more statistically rigorous than the elbow method but computationally expensive because it requires generating many random reference datasets.

Start with domain knowledge — if you know there are 5 product categories, start with k=5. Use the elbow as a quick visual sanity check. Confirm with silhouette for objectivity. Evaluate downstream — for business use cases, test whether the segments are actionable and interpretable. The numerically optimal k is not always the most useful business segmentation.

Here is a compact pipeline that runs both diagnostics side by side, giving you a summary table to help pick k efficiently.

In this lesson you learned: the elbow method plots inertia vs k and looks for the kink where improvement slows, silhouette score ranges from -1 to +1 and measures both cohesion and separation, and combining both methods with domain knowledge gives the most reliable k selection. Next up we explore DBSCAN — a density-based algorithm that discovers clusters of arbitrary shape and handles noise.