Rekursive Feature-Elimination mit Kreuzvalidierung

Univariate feature selection methods like SelectKBest evaluate each feature independently. They miss important cases: a feature that is useless alone but highly valuable combined with another, or two features that are individually strong but highly redundant when combined. Recursive Feature Elimination (RFE) overcomes this by using the model itself to judge feature importance: it fits the model with all features, removes the least important one, refits, removes again, and repeats. This captures how features interact within the model.

RFE works as follows: (1) train the model on all features; (2) rank features by importance (coefficient magnitude for linear models, feature_importances_ for trees); (3) remove the lowest-ranked feature; (4) repeat until the desired number of features remains. At each step, the ranking is recomputed using the model fitted on the remaining features. This means features that appear weak in the presence of redundant competitors might become important once those competitors are removed.

RFECV (Recursive Feature Elimination with Cross-Validation) extends RFE by also determining the optimal number of features to keep. It performs RFE while using cross-validation at each step to score the subset, then selects the number of features that maximises the CV score. This eliminates the need to specify n_features_to_select manually and prevents leakage during selection because the CV loop keeps the test fold isolated.

After fitting, rfecv.support_ is a boolean array indicating which original features were selected. rfecv.ranking_ gives the elimination rank (1 = selected). You can use rfecv.transform(X) to apply the selection to new data, keeping only the columns that survived elimination. This makes RFECV a drop-in replacement for SelectKBest inside Pipelines.

RFECV works with any model that exposes feature_importances_ (tree models) or coef_ (linear models). Using a RandomForestClassifier as the estimator makes RFECV model-agnostic in a powerful way: the forest's own vote on feature importance guides elimination. This is especially useful when the final model is a tree ensemble and you want feature selection to be consistent with the final model's internal ranking.

A subtle leakage risk with RFECV: when you use it with a linear model that requires scaling, you must scale inside the cross-validation loop. If you scale the full dataset before RFECV, the scaler sees the CV test fold during fitting, leaking statistics. The correct approach is to place a StandardScaler before the estimator within RFECV's internal estimator — but RFECV does not directly support Pipeline estimators as its base estimator. An alternative: use a scaling step before RFECV on the training data only, implemented in a custom outer Pipeline.

By default, RFE eliminates one feature per round, requiring N model fits for N features. Setting step to a larger integer or a fraction (e.g., step=0.1 eliminates 10% of remaining features per round) reduces the number of rounds. This is crucial for high-dimensional data: with 1,000 features and step=1, RFE requires 1,000 model fits per CV fold. With step=0.1, roughly 23 rounds suffice. The trade-off is coarser elimination — a group of 100 features is removed together rather than individually.

RFECV and SelectKBest serve different needs. SelectKBest is fast, model-agnostic, and works on univariate correlations — it is a good first pass for large feature sets. RFECV is slower but accounts for feature interactions (how features work together within the model) and automatically selects K. In practice: use SelectKBest to reduce from 1000 to 100 features quickly, then RFECV to refine from 100 to the optimal subset. Combining both is faster than RFECV alone on high-dimensional data.

RFECV is computationally expensive and has limitations. It does not guarantee a globally optimal feature subset — greedy backwards elimination can get stuck in local optima. It is impractical for datasets with thousands of features unless step is large or a fast estimator is used. For very high-dimensional sparse data (NLP, genomics), L1 regularisation (Lasso, LinearSVC with L1) is more efficient because it zeroes out coefficients mathematically without iterative refitting. RFECV shines on medium-dimensional datasets (10-200 features) where iteration is feasible.

Feature selection sits between preprocessing and modelling in the pipeline. The recommended full workflow: (1) clean and encode raw data; (2) apply VarianceThreshold; (3) apply RFECV or SelectKBest inside a Pipeline; (4) train and evaluate with cross-validation; (5) after selection, inspect which features were kept and verify with domain knowledge. Selected features that domain experts cannot explain should be investigated for data leakage — sometimes a feature encodes future information that artificially inflates importance scores.

After RFECV identifies the optimal feature subset, always verify the selection with stakeholders or domain experts. Present the list of selected features and ask: 'Does it make intuitive sense that these features predict the target?' If a feature like 'record ID' or 'timestamp of measurement' appears in the top features, it is almost certainly a data leakage artefact. Conversely, if an obviously important feature (e.g., 'age' for a mortality model) was dropped, investigate why — it may be correlated with another selected feature, or there may be a data quality issue.

Test your understanding of Recursive Feature Elimination from this lesson.

In this lesson you learned: RFE iteratively removes the least important feature as judged by the model itself, capturing feature interactions, RFECV extends RFE by automatically finding the optimal number of features via cross-validation, and the step parameter controls how many features are eliminated per round, trading precision for speed. You have now completed the Feature Engineering course and are ready to explore clustering and unsupervised learning.