From Regression to Classification: Threshold Decisions

Classification is the supervised learning task of predicting which category an input belongs to, rather than predicting a continuous number. Examples include:

• Predicting whether an email is spam or not-spam (binary).
• Predicting which digit (0-9) is in an image (multi-class).
• Predicting which disease a patient has given symptoms (multi-class).

The key distinction from regression: the output is a discrete label, not a real number. This seemingly small change requires different algorithms, different loss functions, and different evaluation metrics.

A tempting approach is to encode class 0 and class 1 as numbers and apply linear regression. For example, encode 'not spam' as 0 and 'spam' as 1, then train a linear regressor. The immediate problem: linear regression produces outputs anywhere from -∞ to +∞, but probabilities must be between 0 and 1. A prediction of 1.7 for class membership is meaningless.

A second problem: linear regression tries to pull the best-fit line through all examples. Adding a clear outlier far from the boundary can rotate the line so it misclassifies many previously-correct examples. The loss function is fundamentally wrong for binary outcomes.

The simplest approach to classification is to use a regression model's output as a score and apply a threshold to convert it to a binary label. If the score is above the threshold, predict class 1; below it, predict class 0.

With a linear regression output, you might choose a threshold of 0.5: anything above 0.5 is spam, below is not-spam. This is called a threshold classifier. While crude, it illustrates the key concept: scores must be converted to decisions, and the choice of threshold involves a trade-off between different types of errors.

Choosing a threshold of 0.5 is arbitrary. The optimal threshold depends on the relative cost of different error types:

• A false positive (predict spam when it is not) means a legitimate email lands in the spam folder.
• A false negative (predict not-spam when it is spam) means junk email reaches the inbox.

In medical diagnosis, the costs are much more asymmetric: a false negative (missing a real disease) may be catastrophic, so you lower the threshold to catch more true positives even at the cost of more false alarms. The threshold is a business decision, not a mathematical one.

Binary classification has exactly two possible output classes (spam/not-spam, disease/healthy, fraud/legitimate). Multi-class classification has three or more classes (which digit 0-9, which species of flower, which product category).

Most binary classifiers extend to multi-class through two strategies:

• One-vs-Rest (OvR): train one binary classifier per class, predict the class whose classifier is most confident.
• One-vs-One (OvO): train a binary classifier for every pair of classes, take a majority vote.

Scikit-learn handles this automatically for most algorithms.

A classifier divides the feature space into regions, one for each class. The boundary between regions is called the decision boundary. For a linear classifier, the decision boundary is a straight line (in 2D), a plane (in 3D), or a hyperplane (in higher dimensions).

The location and shape of the decision boundary is what the algorithm learns during training. Visualising the decision boundary on a 2D dataset is one of the best ways to build intuition for how a classifier works and why its predictions are correct or wrong in specific regions of the feature space.

Accuracy is the fraction of predictions that are correct: accuracy = correct_predictions / total_predictions. It is the most intuitive metric and appropriate when classes are balanced and all errors are equally costly.

However, accuracy is a trap for imbalanced datasets. If 95% of emails are legitimate and your model predicts 'not spam' for everything, it achieves 95% accuracy while completely failing at its job — never catching a single spam email. This is the accuracy paradox, and it motivates more informative metrics like precision and recall.

Most scikit-learn classifiers provide two prediction methods:

• predict(X) — returns the hard class label after applying the default threshold (usually 0.5 for binary classification).
• predict_proba(X) — returns a probability array of shape (n_samples, n_classes), giving the model's confidence in each class.

Using predict_proba gives you much more control because you can apply any threshold. This is essential for business applications where the optimal threshold is not 0.5.

Applying a custom threshold to probability outputs lets you tune the trade-off between false positives and false negatives without retraining the model. This is called threshold moving and is a common post-processing step in production systems.

Lower threshold → model flags more examples as positive → more true positives but also more false positives. Higher threshold → more conservative → fewer false positives but more false negatives. The right threshold is determined by the cost of each error type in your specific application.

Linear regression with a threshold is just the beginning. Scikit-learn provides many dedicated classification algorithms, each with strengths and weaknesses:

• Logistic Regression — the proper probabilistic linear classifier (next lesson).
• K-Nearest Neighbors — classifies by majority vote of nearest training examples.
• Decision Trees — rule-based, fully interpretable.
• Random Forest — ensemble of trees, robust and powerful.
• SVM — finds the maximum-margin hyperplane.
• Naive Bayes — fast, probabilistic, excellent for text.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: linear regression fails for binary classification because it produces unbounded outputs that cannot represent probabilities, applying a threshold converts probability scores to class labels with a tunable trade-off between false positives and false negatives, and accuracy is misleading for imbalanced datasets — recall and precision provide a more complete picture. Next up we study logistic regression — the proper probabilistic linear classifier that uses the sigmoid function to produce calibrated probabilities.