Feature Selection: Variance Threshold and SelectKBest

Adding more features is not always better. Irrelevant features add noise without signal, increasing the risk of overfitting, slowing training, and wasting memory. Redundant features provide duplicate information and can destabilise some models. Feature selection identifies and removes these unhelpful inputs, leaving a compact set of informative features. The result: faster training, lower memory usage, reduced overfitting, and often better generalisation — especially for models without built-in regularisation like KNN or Naive Bayes.

VarianceThreshold removes features whose variance falls below a specified threshold. A feature with zero variance is constant — it has the same value for every example and therefore provides no discriminative information. Near-constant features (very low variance) are almost as uninformative. Setting threshold=0 removes only perfectly constant features; threshold=0.01 removes any feature where 99%+ of values are the same. This is a fast, model-agnostic first pass at cleaning the feature set.

Variance threshold is especially useful after one-hot encoding, when some categories may have very few examples (producing a near-zero-variance binary column) or when working with genomics, text, or sensor data where most features are near-zero. Set the threshold based on the expected proportion of zeros: if 95% of values are zero, the variance is at most 0.95 × 0.05 = 0.0475, so threshold=0.05 removes any feature that is at least 95% constant.

SelectKBest evaluates each feature independently using a univariate statistical test and keeps the top K features. For classification: use chi2 (chi-squared, for non-negative features) or f_classif (ANOVA F-statistic). For regression: use f_regression (linear correlation) or mutual_info_regression. The tests score how much each feature individually correlates with the target. Setting k='all' ranks all features by score rather than selecting a fixed number.

The F-test in f_classif only detects linear relationships between features and target. Mutual information (mutual_info_classif, mutual_info_regression) measures any statistical dependence — linear or non-linear. A feature with a non-linear relationship to the target will score zero with F-test but high with mutual information. Mutual information is slower to compute but more informative, especially for tree-based models where non-linear relationships matter.

How many features should you keep? The right way to choose K is to treat it as a hyperparameter and find the value that maximises cross-validation score. Use SelectKBest inside a Pipeline and add selectkbest__k to your grid search parameters. This avoids the common mistake of selecting K based on training performance (which always improves with more features) and instead finds the K that maximises generalisation.

SelectPercentile is a sibling of SelectKBest that keeps the top percentile% of features rather than a fixed count K. This is more robust when the number of features varies across datasets or when applying the same pipeline to different problems. For example, SelectPercentile(f_classif, percentile=20) keeps the best 20% of features regardless of how many there are. The trade-off: you get a relative budget rather than an absolute count.

Feature selection and regularisation are complementary but different approaches. Feature selection explicitly removes features before training, producing a sparser model with fewer inputs. Regularisation (L1/L2 penalty) keeps all features but shrinks unimportant coefficients toward zero — L1 (Lasso) even produces exact zeros. Feature selection is explicit and interpretable; regularisation is continuous and keeps more information. For interpretability, explicit feature selection is preferred. For predictive power, regularisation often wins by making soft, data-driven importance decisions.

The key limitation of univariate selection (VarianceThreshold, SelectKBest) is that it evaluates each feature independently. A feature that has no marginal correlation with the target might still be very useful in combination with another feature (interaction). Two redundant features might each score high individually but add no value beyond the first one. These methods cannot detect such dependencies. Recursive Feature Elimination (RFECV), covered in the next lesson, addresses this by letting the model itself rank features based on their combined predictive contribution.

A recommended feature selection workflow: (1) Start with VarianceThreshold to remove constant and near-constant features; (2) apply SelectKBest with mutual information for a quick ranking of remaining features; (3) plot CV score vs K to find the elbow; (4) incorporate the selected features in a full pipeline with cross-validation; (5) if the model has built-in importance (tree-based), use permutation importance for a second-pass check. Always embed selection inside a Pipeline to prevent leakage.

SelectFromModel uses a trained model's feature importances (or coefficients) to select features above a threshold. After fitting a random forest or Lasso, pass it to SelectFromModel with a threshold such as 'mean' (keep features with importance above the mean) or 'median'. This is especially useful as a pipeline step before a secondary estimator — it embeds the importance-based selection directly into the cross-validation loop, preventing leakage.

Test your understanding of VarianceThreshold and SelectKBest from this lesson.

In this lesson you learned: VarianceThreshold removes constant and near-constant features as a fast first pass, SelectKBest ranks features by univariate tests (F-test or mutual information) and keeps the top K, and always embed feature selection inside a Pipeline and cross-validate K as a hyperparameter. Next up we explore Recursive Feature Elimination with Cross-Validation (RFECV), which lets the model itself vote out the least useful features.