Recursive Feature Elimination with Cross-Validation
Learners will use RFECV to let the model itself vote out the least useful features while preserving the cross-validation loop to prevent leakage.
Recursive Feature Elimination with Cross-Validation is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Limitations of Univariate Selection
Univariate feature selection methods like SelectKBest evaluate each feature independently. They miss important cases: a feature that is useless alone but highly valuable combined with another, or two features that are individually strong but highly redundant when combined. Recursive Feature Elimination (RFE) overcomes this by using the model itself to judge feature importance: it fits the model with all features, removes the least important one, refits, removes again, and repeats. This captures how features interact within the model.
How RFE Works
RFE works as follows: (1) train the model on all features; (2) rank features by importance (coefficient magnitude for linear models, feature_importances_ for trees); (3) remove the lowest-ranked feature; (4) repeat until the desired number of features remains. At each step, the ranking is recomputed using the model fitted on the remaining features. This means features that appear weak in the presence of redundant competitors might become important once those competitors are removed.
from sklearn.feature_selection import RFE
from sklearn.svm import SVR
from sklearn.datasets import load_diabetes
import numpy as np
X, y = load_diabetes(return_X_y=True)
feature_names = load_diabetes().feature_names
# Select top 5 features
rfe = RFE(estimator=SVR(kernel='linear'), n_features_to_select=5)
rfe.fit(X, y)
print('Selected features:', [feature_names[i] for i in range(len(feature_names)) if rfe.support_[i]])
print('Feature rankings:', rfe.ranking_) # 1 = selectedRFECV: Choosing the Number of Features Automatically
RFECV (Recursive Feature Elimination with Cross-Validation) extends RFE by also determining the optimal number of features to keep. It performs RFE while using cross-validation at each step to score the subset, then selects the number of features that maximises the CV score. This eliminates the need to specify n_features_to_select manually and prevents leakage during selection because the CV loop keeps the test fold isolated.
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = load_breast_cancer(return_X_y=True)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # scale first, then RFECV
rfecv = RFECV(
estimator=LogisticRegression(max_iter=1000),
step=1,
cv=5,
scoring='accuracy',
n_jobs=-1
)
rfecv.fit(X_scaled, y)
print('Optimal number of features:', rfecv.n_features_)
print('CV scores per feature count:', rfecv.cv_results_['mean_test_score'].round(4))RFECV and the Support Mask
After fitting, rfecv.support_ is a boolean array indicating which original features were selected. rfecv.ranking_ gives the elimination rank (1 = selected). You can use rfecv.transform(X) to apply the selection to new data, keeping only the columns that survived elimination. This makes RFECV a drop-in replacement for SelectKBest inside Pipelines.
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
import numpy as np
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
X_sc = StandardScaler().fit_transform(X)
rfecv = RFECV(LogisticRegression(max_iter=1000), cv=5, scoring='accuracy', n_jobs=-1)
rfecv.fit(X_sc, y)
selected = [feature_names[i] for i in range(len(feature_names)) if rfecv.support_[i]]
print(f'Selected {len(selected)} features:', selected)Using RFE with Tree-Based Models
RFECV works with any model that exposes feature_importances_ (tree models) or coef_ (linear models). Using a RandomForestClassifier as the estimator makes RFECV model-agnostic in a powerful way: the forest's own vote on feature importance guides elimination. This is especially useful when the final model is a tree ensemble and you want feature selection to be consistent with the final model's internal ranking.
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
rfecv = RFECV(
estimator=RandomForestClassifier(n_estimators=50, random_state=42),
step=1,
cv=5,
scoring='accuracy',
n_jobs=-1
)
rfecv.fit(X, y) # RF does not require scaling
print('Optimal features:', rfecv.n_features_)
print('Best CV score:', round(max(rfecv.cv_results_['mean_test_score']), 4))RFECV Inside a Pipeline: Preventing Leakage
A subtle leakage risk with RFECV: when you use it with a linear model that requires scaling, you must scale inside the cross-validation loop. If you scale the full dataset before RFECV, the scaler sees the CV test fold during fitting, leaking statistics. The correct approach is to place a StandardScaler before the estimator within RFECV's internal estimator — but RFECV does not directly support Pipeline estimators as its base estimator. An alternative: use a scaling step before RFECV on the training data only, implemented in a custom outer Pipeline.
Step Parameter: Faster but Coarser
By default, RFE eliminates one feature per round, requiring N model fits for N features. Setting step to a larger integer or a fraction (e.g., step=0.1 eliminates 10% of remaining features per round) reduces the number of rounds. This is crucial for high-dimensional data: with 1,000 features and step=1, RFE requires 1,000 model fits per CV fold. With step=0.1, roughly 23 rounds suffice. The trade-off is coarser elimination — a group of 100 features is removed together rather than individually.
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
import time
X, y = load_breast_cancer(return_X_y=True)
X_sc = StandardScaler().fit_transform(X)
for step in [1, 2, 5]:
start = time.time()
rfecv = RFECV(LogisticRegression(max_iter=1000), step=step, cv=5, n_jobs=-1)
rfecv.fit(X_sc, y)
print(f'step={step}: {round(time.time()-start,2)}s, optimal_features={rfecv.n_features_}')Comparing RFECV with SelectKBest
RFECV and SelectKBest serve different needs. SelectKBest is fast, model-agnostic, and works on univariate correlations — it is a good first pass for large feature sets. RFECV is slower but accounts for feature interactions (how features work together within the model) and automatically selects K. In practice: use SelectKBest to reduce from 1000 to 100 features quickly, then RFECV to refine from 100 to the optimal subset. Combining both is faster than RFECV alone on high-dimensional data.
from sklearn.feature_selection import SelectKBest, f_classif, RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
X_sc = StandardScaler().fit_transform(X)
# Baseline: no selection
base = cross_val_score(LogisticRegression(max_iter=1000), X_sc, y, cv=5).mean()
# SelectKBest
X_skb = SelectKBest(f_classif, k=15).fit_transform(X_sc, y)
skb = cross_val_score(LogisticRegression(max_iter=1000), X_skb, y, cv=5).mean()
# RFECV
rfecv = RFECV(LogisticRegression(max_iter=1000), cv=5, n_jobs=-1).fit(X_sc, y)
rfe_score = rfecv.cv_results_['mean_test_score'].max()
print(f'All features: {base:.4f}')
print(f'SelectKBest(k=15): {skb:.4f}')
print(f'RFECV: {rfe_score:.4f} ({rfecv.n_features_} features)')When RFECV Is Not the Best Choice
RFECV is computationally expensive and has limitations. It does not guarantee a globally optimal feature subset — greedy backwards elimination can get stuck in local optima. It is impractical for datasets with thousands of features unless step is large or a fast estimator is used. For very high-dimensional sparse data (NLP, genomics), L1 regularisation (Lasso, LinearSVC with L1) is more efficient because it zeroes out coefficients mathematically without iterative refitting. RFECV shines on medium-dimensional datasets (10-200 features) where iteration is feasible.
Feature Selection in the Full ML Workflow
Feature selection sits between preprocessing and modelling in the pipeline. The recommended full workflow: (1) clean and encode raw data; (2) apply VarianceThreshold; (3) apply RFECV or SelectKBest inside a Pipeline; (4) train and evaluate with cross-validation; (5) after selection, inspect which features were kept and verify with domain knowledge. Selected features that domain experts cannot explain should be investigated for data leakage — sometimes a feature encodes future information that artificially inflates importance scores.
Verifying Selected Features with Domain Experts
After RFECV identifies the optimal feature subset, always verify the selection with stakeholders or domain experts. Present the list of selected features and ask: 'Does it make intuitive sense that these features predict the target?' If a feature like 'record ID' or 'timestamp of measurement' appears in the top features, it is almost certainly a data leakage artefact. Conversely, if an obviously important feature (e.g., 'age' for a mortality model) was dropped, investigate why — it may be correlated with another selected feature, or there may be a data quality issue.
Quick Check
Test your understanding of Recursive Feature Elimination from this lesson.
Lesson Recap
In this lesson you learned: RFE iteratively removes the least important feature as judged by the model itself, capturing feature interactions, RFECV extends RFE by automatically finding the optimal number of features via cross-validation, and the step parameter controls how many features are eliminated per round, trading precision for speed. You have now completed the Feature Engineering course and are ready to explore clustering and unsupervised learning.
Frequently asked questions
Is the “Recursive Feature Elimination with Cross-Validation” lesson free?
Yes — the full text of “Recursive Feature Elimination with Cross-Validation” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Recursive Feature Elimination with Cross-Validation”?
Learners will use RFECV to let the model itself vote out the least useful features while preserving the cross-validation loop to prevent leakage. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Recursive Feature Elimination with Cross-Validation” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Creating New Features: Log Transforms, Binning, and Interactions
- Date and Time Feature Extraction
- Feature Selection: Variance Threshold and SelectKBest
- Recursive Feature Elimination with Cross-Validation