Creating New Features: Log Transforms, Binning, and Interactions

Feature engineering is the process of transforming raw data into representations that make patterns more accessible to machine learning models. Even the best algorithm is limited by the quality of its input features. Adding the right engineered feature can boost model accuracy more than any amount of hyperparameter tuning. The intuition: if you give the model the right numbers to work with, it can learn simpler, more generalisable rules than if it must discover complex transformations by itself.

Many real-world quantities — income, house prices, population, transaction amounts — follow right-skewed distributions where most values are small but a few extreme values stretch the tail. Linear models and distance-based algorithms (KNN, SVM) perform poorly on such features because the large values dominate distance calculations. Applying np.log1p() (log of x+1, safe for zero values) compresses the scale, making the distribution more symmetric and reducing the influence of extreme outliers.

The benefit of log-transforming skewed features is not just visual — it directly improves model performance for algorithms that assume normally-distributed features (linear/logistic regression) or that use distances (KNN, SVM). For tree-based models (decision trees, random forests, gradient boosting), the benefit is smaller because trees split on thresholds and are naturally scale-invariant. Always verify the improvement with cross-validation rather than assuming the transform helps.

Binning (or bucketing) converts a continuous feature into discrete categories. For example, age [0-100] might be binned into [child, teen, adult, senior]. This can help when the relationship between a feature and the target is non-linear in a step-function way — the model learns one coefficient per bin rather than trying to fit a linear slope. Pandas pd.cut() uses equal-width bins; pd.qcut() uses equal-frequency (quantile) bins that put the same number of examples in each bin.

For use in scikit-learn Pipelines, KBinsDiscretizer provides the same functionality with the standard fit/transform API. The strategy parameter controls bin edges: 'uniform' (equal width), 'quantile' (equal frequency), or 'kmeans' (k-means clustering on the feature). The encode parameter controls output format: 'onehot' (one-hot sparse matrix), 'onehot-dense', or 'ordinal' (integer labels).

Interaction features are products or ratios of existing features that capture effects that neither feature alone can express. For example, rooms_per_person = total_rooms / population captures housing density better than either feature individually. A linear model cannot discover that room count × house age matters; you must create that product explicitly. Domain knowledge is invaluable here — ask 'what combination of these numbers would a domain expert find meaningful?'

PolynomialFeatures from scikit-learn automates interaction and polynomial term generation. With degree=2, it creates all pairwise products and squared terms. For p original features, it generates p(p+1)/2 interaction features plus p squared features. This is powerful for linear models on low-dimensional data, but becomes computationally prohibitive for high-dimensional data (100 features → 5,050 interaction terms). Always follow with feature selection or regularisation when using polynomial features.

Ratio features normalise raw counts by a relevant denominator, removing scale effects. Examples: crime_rate = crimes / population, revenue_per_user = revenue / active_users, defect_rate = defects / total_units. Ratios are informative when the numerator and denominator vary independently and the ratio captures a meaningful rate that neither alone captures. Always guard against division by zero using .clip(lower=epsilon) or adding a small constant.

For high-cardinality categorical features (hundreds of unique values), one-hot encoding creates too many dimensions. Target encoding replaces each category with the mean of the target for that category. For example, the 'city' feature gets replaced by the average sale price in each city. This is powerful but prone to leakage if done naively — the target encoding must be computed on the training data only and applied to the test fold. Use TargetEncoder from scikit-learn inside a Pipeline for correct implementation.

The right way to evaluate whether a new feature helps: (1) start with a baseline CV score on the original features; (2) add the new feature inside the training pipeline; (3) compare CV scores. Improvements of >0.5% on a robust metric (5-fold CV AUC or RMSE) are worth keeping. Features that hurt or show no improvement should be dropped — irrelevant features add noise and slow down training. Use feature importance (from tree models) or permutation importance to identify which engineered features are actually used.

Feature engineering can be done manually (using domain knowledge to craft specific features) or automatically (using tools like featuretools for deep feature synthesis or scikit-learn's PolynomialFeatures). Manual, domain-guided features typically outperform automated approaches because they embed human understanding of what the data actually means. Automated tools explore a much larger feature space and may find unexpected interactions, but also generate many useless features that require selection. The best practice is to start with domain-guided features, then optionally add automated candidates and filter them with feature selection.

Test your understanding of Feature Engineering from this lesson.

In this lesson you learned: log transforms reduce skew and make linear models work better on exponentially-distributed features, binning discretises continuous features into categories that capture step-wise relationships, and interaction features capture multiplicative effects that neither feature alone can express. Next up we explore extracting features from date and time columns.