Multiple Linear Regression and Feature Importance

Real prediction problems almost always have multiple relevant inputs. House price depends on square footage, number of bedrooms, location, age, and dozens of other factors. Multiple linear regression extends the single-feature line to use all available features simultaneously:

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

Each feature gets its own weight, and the model learns all weights simultaneously by minimising MSE. Adding more relevant features almost always improves prediction accuracy — as long as those features genuinely carry information about the target.

The scikit-learn API is identical for one feature or one hundred features — you simply pass the full feature matrix. The model automatically fits one coefficient per column in X.

With multiple features, R² typically increases compared to the single-feature model, because each additional informative feature provides the model with more signal to work with. However, adding irrelevant or noisy features can actually hurt performance by introducing overfitting, especially on small datasets.

After fitting a multi-feature model, model.coef_ is an array with one value per feature. Each value is the predicted change in y for a one-unit increase in that feature, assuming all other features are held constant.

For example, a coefficient of 150 for sqft means: controlling for bedrooms, age, and all other features, one additional square foot is predicted to add $150 to the price. This partial effect interpretation is what makes linear regression so useful for explaining which factors drive a prediction.

Raw coefficients are not directly comparable when features have different units and scales. In the housing model, sqft ranges from 600 to 4000 while bathrooms ranges from 1 to 5. A coefficient of 150 for sqft and 8000 for bedrooms does not mean bedrooms are 53x more important — it means a one-unit change in bedrooms (a huge change) has 53x the effect of a one-unit change in sqft (a tiny change).

To compare feature importance fairly, you must first standardise all features to the same scale (zero mean, unit variance). Then the coefficients reflect the effect of a one-standard-deviation change in each feature, which is comparable across features.

Visualising standardised coefficients as a bar chart gives stakeholders an intuitive view of which features drive the model's predictions. Positive bars push the prediction up; negative bars push it down. The longer the bar, the greater the impact of a one-standard-deviation change in that feature.

This plot is one of the most persuasive tools for communicating model behaviour to non-technical audiences. It answers 'what matters?' in a single, easy-to-read chart.

Regular R² always increases (or stays the same) when you add more features, even if those features are random noise. This makes it a misleading metric for model selection with different numbers of features.

Adjusted R² corrects for this by penalising for each additional feature: Adj R² = 1 - (1-R²)(n-1)/(n-p-1), where n is the number of samples and p is the number of features. If an added feature does not improve the model enough to justify its complexity, Adjusted R² decreases. Use Adjusted R² when comparing models with different numbers of features.

Multicollinearity occurs when two or more features are highly correlated with each other. For example, sqft and number_of_rooms often move together — bigger houses have more rooms. When features are highly correlated, the model cannot reliably estimate individual coefficients because it cannot distinguish which feature is actually responsible for the effect on y.

Symptoms include: coefficients that change dramatically when you add or remove a single feature, very large coefficient values with opposite signs, or high R² but all coefficients with large standard errors. The Variance Inflation Factor (VIF) is the standard diagnostic for multicollinearity.

Not all features improve model performance. Adding irrelevant features increases complexity and can hurt generalisation. A disciplined approach to feature selection involves comparing models with different subsets of features using cross-validated Adjusted R².

Iterative forward selection adds one feature at a time, keeping it only if it improves the validation score. Recursive feature elimination (RFE) removes the least important feature iteratively. Scikit-learn provides both approaches.

Once you have a trained multi-feature model, making predictions on new data is straightforward: create a DataFrame or array with the same feature columns in the same order, and call model.predict(). The model returns one predicted price per row.

If you used a Pipeline with a scaler, pass the raw (unscaled) data — the pipeline automatically applies the same scaling transformation that was fitted on training data. This is another reason why Pipelines are the professional way to work.

Linear regression is limited to linear relationships, but you can model non-linear patterns by engineering polynomial features. PolynomialFeatures in scikit-learn automatically creates squared, cubed, and interaction terms from your original features.

A degree-2 polynomial on two features x₁ and x₂ produces: x₁, x₂, x₁², x₂², x₁x₂. The model is still linear in its parameters (which is why it is still 'linear regression'), but it can fit curved relationships in the original feature space. Use with cross-validation to pick the optimal degree.

Key takeaways for using coefficients to measure feature importance:

• Raw coefficients depend on feature scale — a coefficient of 150 for sqft is not directly comparable to 8000 for bedrooms unless both are standardised.
• Standardised coefficients (from a model trained on scaled features) measure the effect of a one-standard-deviation change in each feature, making them directly comparable.
• Large absolute values indicate influential features; small values (near zero) indicate weak or redundant features.
• Always confirm that high-importance features have a causal or meaningful relationship with the target, not just statistical correlation in the training data.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: multiple linear regression assigns one weight per feature in the equation ŷ = w₁x₁ + ... + wₙxₙ + b, raw coefficients must be standardised before comparing importance across features with different scales, and Adjusted R² penalises added features to prevent overfitting during model selection. Next up we cross the boundary from predicting numbers to predicting categories with logistic regression and threshold-based classification.