Maximum Margin Classifier: Support Vectors and Hyperplane

Support Vector Machines (SVMs) are classifiers that find the best separating boundary between two classes. When multiple boundaries can separate the classes, which one should you choose? The SVM's answer is elegant: pick the boundary that is as far as possible from every training example. This maximally distant boundary is called the maximum-margin hyperplane, and SVM's theoretical foundations guarantee that it generalises better to unseen data than arbitrary separating boundaries.

In 2D, a hyperplane is a line (1 dimension less than the data space). In 3D, it is a plane. In general p-dimensional space, it is a (p-1)-dimensional flat surface defined by the equation w·x + b = 0, where w is the normal vector (perpendicular to the surface), x is the input feature vector, and b is the bias term. Points on one side satisfy w·x + b > 0 (predicted positive class) and points on the other satisfy w·x + b < 0 (predicted negative class).

The margin is the distance between the decision boundary and the closest training examples from each class. The SVM defines two margin hyperplanes parallel to the decision boundary: w·x + b = +1 for the positive class boundary and w·x + b = -1 for the negative class boundary. The total margin width is 2 / ||w||. To maximise the margin, the SVM minimises ||w|| (equivalently, ||w||²/2 for mathematical convenience) subject to the constraint that all points are correctly classified.

Support vectors are the training examples that lie exactly on the margin hyperplanes (where w·x + b = ±1). They are the only examples that determine the position and orientation of the decision boundary. All other training examples — those farther from the boundary — play no role in defining it. This is a profound insight: the SVM decision boundary is entirely defined by a small subset of the training data, making it robust to the majority of the training set.

Use sklearn.svm.SVC with kernel='linear' for a linear maximum-margin classifier. After fitting, the decision function score for a point is its signed distance to the decision boundary — positive for class 1, negative for class 0. The decision_function() method returns these raw scores, while predict() applies the sign threshold to produce class labels.

SVMs compute distances between points and the hyperplane using the dot product w·x. If one feature ranges from 0 to 1 and another from 0 to 1,000,000, the large-scale feature will dominate the distance calculation, causing the SVM to almost ignore the small-scale feature. Always apply StandardScaler (or MinMaxScaler) before training an SVM. This is one of the most common mistakes beginners make — even a perfect hyperplane can fail if features are not scaled.

Imagine placing a road between two rows of trees (classes). The road's center is the decision boundary, and its width is the margin. You want to build the widest road that still fits between the trees without hitting any of them. The trees closest to the road are the support vectors. A wider road is better because it has more tolerance — a new tree can be placed anywhere within the road's width and still be on the correct side of the boundary.

SVMs can be trained in two equivalent ways: the primal form (optimise over w and b directly) and the dual form (optimise over a set of Lagrange multipliers, one per training example). The dual form is significant because the optimisation only involves dot products between training examples. Replacing these dot products with a kernel function implicitly maps data to a higher-dimensional space without computing the coordinates explicitly — this is the famous kernel trick that enables non-linear SVMs.

The SVM decision_function() returns the signed distance from each point to the decision hyperplane. Points with large positive scores are confidently in the positive class; large negative scores indicate the negative class. Points near zero are close to the boundary and represent the most uncertain predictions. Monitoring the distribution of decision function scores on a new dataset is a useful diagnostic — if most scores cluster near zero, the model may be poorly suited to the data.

The maximum-margin formulation described so far is the hard margin SVM, which requires perfect linear separability — no training point can violate the margin. Real-world data is almost never perfectly linearly separable due to noise and overlapping class distributions. Applying a hard-margin SVM to such data will fail (the optimisation has no feasible solution). The practical solution is the soft-margin SVM, introduced in the next lesson, which allows some margin violations controlled by a penalty parameter C.

The basic SVM formulation handles binary classification. For multi-class problems, scikit-learn's SVC uses a one-vs-one strategy by default: it trains k(k-1)/2 binary classifiers, one for each pair of classes, and predicts the class that wins the most pairwise votes. For 10 classes, this means 45 binary classifiers. An alternative is one-vs-rest (via LinearSVC), which trains k binary classifiers, each distinguishing one class from all others. One-vs-one is generally more accurate but slower for many classes.

Test your understanding of the Maximum Margin Classifier from this lesson.

In this lesson you learned: SVMs find the maximum-margin hyperplane separating two classes, support vectors are the critical examples on the margin that define the boundary, and feature scaling is essential before training an SVM. Next up we explore the soft-margin SVM and the C parameter that allows controlled margin violations.