Feature Selection Methods
Filter (variance, correlation), wrapper (RFE), embedded (L1 regularization) methods.
Feature Selection Methods is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Select Features
Feature selection keeps only the most useful columns. Fewer features mean faster training, less overfitting, and easier interpretation. Removing noise often improves accuracy too.
Three Families of Methods
Selection methods fall into three groups:
- Filter rank features by a statistic (fast, model-free)
- Wrapper search subsets by training models (slow, accurate)
- Embedded selection happens during model fitting
VarianceThreshold
The simplest filter: VarianceThreshold drops features whose variance is below a cutoff. Near-constant columns carry no information.
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=0.01)
X_reduced = sel.fit_transform(X)
print("Kept:", X_reduced.shape[1], "of", X.shape[1])SelectKBest with f_classif
SelectKBest keeps the K highest-scoring features. With f_classif it uses an ANOVA F-test measuring how strongly each feature relates to the class label.
from sklearn.feature_selection import SelectKBest, f_classif
sel = SelectKBest(score_func=f_classif, k=10)
X_best = sel.fit_transform(X, y)
print(sel.get_support(indices=True))Mutual Information
mutual_info_classif measures any (including non-linear) dependency between a feature and the target. Unlike the F-test, it captures non-linear relationships that ANOVA misses.
from sklearn.feature_selection import SelectKBest, mutual_info_classif
sel = SelectKBest(score_func=mutual_info_classif, k=10)
X_mi = sel.fit_transform(X, y)f_classif vs mutual_info Comparison
f_classif is fast and detects linear relationships; mutual_info_classif is slower but catches non-linear ones. If you suspect complex relationships, prefer mutual information.
import numpy as np
from sklearn.feature_selection import f_classif, mutual_info_classif
f_scores, _ = f_classif(X, y)
mi_scores = mutual_info_classif(X, y)
print("F-test top:", np.argsort(f_scores)[::-1][:5])
print("MI top:", np.argsort(mi_scores)[::-1][:5])Recursive Feature Elimination
RFE is a wrapper method: it trains a model, removes the weakest feature, and repeats. It uses the model own importance signal to rank features.
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
rfe = RFE(estimator=LogisticRegression(max_iter=1000), n_features_to_select=10)
rfe.fit(X, y)
print(rfe.support_)RFECV: Choosing the Count Automatically
RFECV adds cross-validation to RFE so it finds the optimal number of features by validation score, not a fixed guess.
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
rfecv = RFECV(
estimator=RandomForestClassifier(),
cv=5,
scoring="accuracy",
)
rfecv.fit(X, y)
print("Optimal features:", rfecv.n_features_)SelectFromModel with Lasso
Embedded selection: Lasso (L1) shrinks unimportant coefficients to exactly zero. SelectFromModel then keeps only non-zero ones.
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import Lasso
sel = SelectFromModel(Lasso(alpha=0.01))
sel.fit(X, y)
X_lasso = sel.transform(X)
print("Kept:", X_lasso.shape[1])SelectFromModel with Tree Importances
SelectFromModel also works with any estimator exposing feature_importances_, like random forests. Set a threshold such as "mean" or "median".
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier
sel = SelectFromModel(
RandomForestClassifier(n_estimators=200),
threshold="median",
)
sel.fit(X, y)Choosing a Method
Start with cheap filters (VarianceThreshold, SelectKBest) to drop obvious junk. Use embedded SelectFromModel for a good balance. Reserve RFECV for when accuracy matters most and compute allows.
Quick Check
Test your feature selection knowledge.
Recap
Recap: Feature selection has filter (VarianceThreshold, SelectKBest with f_classif or mutual_info_classif), wrapper (RFECV), and embedded (SelectFromModel with Lasso or tree importances) methods. Filters are fastest; RFECV is most accurate. Use mutual information for non-linear relationships.
Frequently asked questions
Is the “Feature Selection Methods” lesson free?
Yes — the full text of “Feature Selection Methods” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Feature Selection Methods”?
Filter (variance, correlation), wrapper (RFE), embedded (L1 regularization) methods. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Feature Selection Methods” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.