From Regression to Classification: Threshold Decisions
Learners will understand why linear regression fails for binary outcomes and see how adding a threshold converts a score into a class label.
What Is Classification?
Classification is the supervised learning task of predicting which category an input belongs to, rather than predicting a continuous number. Examples include:
- Predicting whether an email is spam or not-spam (binary).
- Predicting which digit (0-9) is in an image (multi-class).
- Predicting which disease a patient has given symptoms (multi-class).
The key distinction from regression: the output is a discrete label, not a real number. This seemingly small change requires different algorithms, different loss functions, and different evaluation metrics.
Why Linear Regression Fails for Classification
A tempting approach is to encode class 0 and class 1 as numbers and apply linear regression. For example, encode 'not spam' as 0 and 'spam' as 1, then train a linear regressor. The immediate problem: linear regression produces outputs anywhere from -∞ to +∞, but probabilities must be between 0 and 1. A prediction of 1.7 for class membership is meaningless.
A second problem: linear regression tries to pull the best-fit line through all examples. Adding a clear outlier far from the boundary can rotate the line so it misclassifies many previously-correct examples. The loss function is fundamentally wrong for binary outcomes.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
# Binary dataset: 1 = spam, 0 = not spam
word_count = np.array([10, 20, 25, 30, 40, 50, 200]).reshape(-1, 1)
spam = np.array([0, 0, 0, 1, 1, 1, 1])
model = LinearRegression()
model.fit(word_count, spam)
# Problematic: predictions outside [0,1]
predictions = model.predict([[5], [25], [200]])
print('Linear regression predictions (should be 0 or 1):')
print(predictions) # might be negative or >1All lessons in this course
- From Regression to Classification: Threshold Decisions
- Logistic Regression and the Sigmoid Function
- The Confusion Matrix Explained
- Precision, Recall, and F1-Score in Practice