Regression Metrics: MAE, MSE, RMSE, and R-Squared

Regression models predict continuous values — house prices, temperatures, sales volumes. Unlike classification, where a prediction is either correct or wrong, regression predictions are almost never exactly right. The question is: how far off is the prediction? Regression metrics quantify the difference between predicted values and actual values. Different metrics weight errors differently: some penalise large errors more than small ones, some are robust to outliers, and some are normalised to be interpretable as a fraction of variance explained. Choosing the right metric depends on the problem's error cost structure.

MAE = mean(|y_true - y_pred|). Mean Absolute Error computes the average of the absolute differences between predictions and actual values. It is in the same units as the target variable — a house price MAE of $15,000 means predictions are off by $15,000 on average. MAE treats all errors linearly: a prediction that is twice as wrong costs exactly twice as much. MAE is robust to outliers because it does not square errors. It is the recommended metric when the distribution of errors is roughly symmetric and outlier errors should not dominate the aggregate.

MSE = mean((y_true - y_pred)^2). Mean Squared Error squares each error before averaging. This makes MSE sensitive to outliers: a single prediction with error 10 contributes 100 to MSE, while a prediction with error 1 contributes only 1 — outlier errors are amplified quadratically. MSE is the natural loss function for linear regression (minimising MSE gives the ordinary least squares solution). Its units are the square of the target variable (dollars^2 for house prices), making it less interpretable than MAE but mathematically convenient for optimisation.

RMSE = sqrt(MSE). Root Mean Squared Error brings MSE back to the original units of the target variable (dollars, not dollars^2) by taking the square root. It is the most widely reported regression metric because it is interpretable in the same units as the target while still penalising large errors more than small ones. An RMSE of $20,000 for house prices means the typical error magnitude is $20,000, though outlier errors are weighted more heavily than in MAE. RMSE is always >= MAE for the same predictions; the gap between them increases when outlier errors are present.

Choose MAE when: outliers are expected and you do not want them to dominate the metric; you want a metric that is easily interpretable as the average error; errors of different sizes are equally important proportionally. Choose RMSE when: large errors are much more costly than small ones (e.g., predicting power demand — being off by 10x is far more costly than being off by 1x); the loss function used in training is MSE and you want evaluation to match training; variance in predictions matters more than median error. In practice, always report both MAE and RMSE — if RMSE >> MAE, outlier errors dominate; if RMSE ≈ MAE, errors are uniformly distributed.

R^2 = 1 - MSE(model) / Var(y_true). R-squared measures the fraction of variance in the target explained by the model. R^2 = 1.0 means perfect predictions (zero residual variance). R^2 = 0.0 means the model is no better than predicting the mean of y for every sample. R^2 can be negative if the model is worse than the mean baseline. R^2 is scale-independent — a value of 0.85 means the same thing whether the target is house prices or temperatures. This makes it the primary metric for comparing regression models across different problems or scales.

Adding more features to a regression model always increases R^2 (or keeps it the same) on training data — even if those features are random noise. Adjusted R^2 corrects for this by penalising the number of features: it decreases when you add a feature that does not improve the model enough to justify the additional complexity. Formula: Adj R^2 = 1 - (1 - R^2) * (n-1) / (n-p-1) where n = samples, p = features. Use adjusted R^2 when comparing models with different numbers of features to avoid selecting overly complex models.

MAPE = 100 * mean(|y_true - y_pred| / |y_true|). Mean Absolute Percentage Error expresses the error as a percentage of the actual value, making it scale-independent and easily interpretable: a MAPE of 5% means predictions are off by 5% of the true value on average. Limitations: MAPE is undefined when actual values are zero (division by zero) and is asymmetric — underpredicting gives higher error than overpredicting by the same amount. Use MAPE for forecasting tasks (sales, demand) where stakeholders understand percentage deviations naturally.

Scikit-learn provides all major regression metrics in sklearn.metrics. When using cross_val_score, pass scoring='neg_mean_squared_error' (negative because higher is better by convention) and negate the result, or use 'neg_root_mean_squared_error' for direct RMSE. For R^2, use scoring='r2'. Always report multiple metrics: MAE for average absolute error, RMSE to show outlier sensitivity, and R^2 for explained variance. This gives a complete picture of model performance.

Beyond summary metrics, residual analysis reveals whether model assumptions are met. Plot residuals (y_true - y_pred) vs predicted values. A good model shows: (1) residuals randomly scattered around zero with no pattern (homoscedastic), (2) no relationship between residual magnitude and predicted value (no heteroscedasticity). Warning signs: a funnel shape (residuals increase with predicted value) suggests the model needs log-transformation; a curved pattern suggests the relationship is non-linear and the model is misspecified. Always do residual analysis before deploying any regression model.

A practical guide: use MAE when you want to minimise average absolute error and outliers should not be overemphasised — robust choice for general reporting. Use RMSE when your model minimises MSE during training and you want evaluation to match, or when large errors are particularly costly (demand forecasting, energy prediction). Use R^2 to compare models on the same dataset or explain model quality to non-technical stakeholders as 'percentage of variance explained'. Use MAPE for business forecasting where percentage deviations from actual values are the natural language of stakeholders.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: MAE averages absolute errors and is robust to outliers, RMSE squares errors and penalises large deviations more heavily, R^2 measures the fraction of variance explained (0 = mean baseline, 1 = perfect), and the gap between MAE and RMSE reveals outlier error concentration. Next up we explore choosing the right metric for your specific business problem.