Feature Engineering and Selection

Feature engineering and selection are crucial steps in building effective machine learning models. These processes involve creating new features and selecting the most relevant ones to improve model performance.

In this lesson, you’ll learn techniques for feature engineering and selection to enhance predictive accuracy.

What is Feature Engineering?

Feature engineering involves creating new features from raw data to make machine learning models more effective. Examples include:

• Extracting time-based features like day, month, or hour from timestamps.
• Combining multiple features into a new feature.
• Scaling numerical features to standardize their range.

Creating New Features

Let’s create new features based on existing data:

# Example: Creating new features
import pandas as pd

data = {'Timestamp': ['2025-01-01 12:00:00', '2025-01-02 15:30:00']}
df = pd.DataFrame(data)
df['Hour'] = pd.to_datetime(df['Timestamp']).dt.hour
df['Day'] = pd.to_datetime(df['Timestamp']).dt.day
print(df)

Scaling Features

Scaling ensures all numerical features are on the same scale, which improves model performance:

# Example: Scaling features
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
values = [[10], [20], [30]]
scaled_values = scaler.fit_transform(values)
print(scaled_values)

Feature Selection

Feature selection involves choosing the most relevant features for the model. Common techniques include:

• Filter Methods: Selecting features based on statistical measures like correlation.
• Wrapper Methods: Using algorithms like Recursive Feature Elimination (RFE).
• Embedded Methods: Selecting features during model training, e.g., Lasso regression.

Correlation-Based Feature Selection

Highly correlated features can be redundant. Use Pandas to compute correlations:

# Example: Correlation-based feature selection
import pandas as pd

data = {'Feature1': [1, 2, 3], 'Feature2': [2, 4, 6], 'Feature3': [1, 1, 1]}
df = pd.DataFrame(data)
print(df.corr())

Recursive Feature Elimination (RFE)

RFE is a wrapper method that iteratively removes the least important features:

# Example: RFE
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

X = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
y = [0, 1, 0]
model = LogisticRegression()
rfe = RFE(model, n_features_to_select=2)
rfe.fit(X, y)
print("Selected Features:", rfe.support_)

Feature Importance with Tree-Based Models

Tree-based models like Random Forests provide feature importance scores:

# Example: Feature importance
from sklearn.ensemble import RandomForestClassifier

X = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
y = [0, 1, 0]
model = RandomForestClassifier()
model.fit(X, y)
print("Feature Importances:", model.feature_importances_)

Common Mistakes in Feature Engineering

Here are some mistakes to avoid:

• Creating too many features, which can lead to overfitting.
• Ignoring the need for scaling in distance-based algorithms.
• Using highly correlated features, causing redundancy.

What Did We Learn?

In this lesson, you learned:

• The importance of feature engineering and selection in machine learning.
• Techniques for creating new features and scaling numerical features.
• How to select relevant features using methods like RFE and tree-based models.
• Common mistakes to avoid during feature engineering.

Great job! Let’s move to the next topic.