Combining Steps with ColumnTransformer
Learners will apply different preprocessing steps to different column types simultaneously using ColumnTransformer, producing a single clean feature matrix.
Combining Steps with ColumnTransformer is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Problem ColumnTransformer Solves
Real datasets contain a mix of column types: some are numeric, some are ordinal, and some are nominal categorical. Applying the same transformer to all columns would be incorrect — you cannot StandardScale a one-hot column or OrdinalEncode a continuous number. Before ColumnTransformer, data scientists had to manually slice DataFrames, apply different transformers, and re-concatenate — a fragile, leakage-prone process. ColumnTransformer solves this by applying different transformations to different column subsets simultaneously inside a single, pipeline-compatible object.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'age': [25, 35, 45, 28],
'income': [40000, 80000, 120000, 55000],
'education': ['bachelor', 'master', 'PhD', 'high school'],
'city': ['NYC', 'LA', 'NYC', 'Chicago']
})
# Need: StandardScaler for age+income
# OrdinalEncoder for education
# OneHotEncoder for city
# ColumnTransformer applies all three at onceColumnTransformer Syntax and Structure
A ColumnTransformer is constructed as a list of (name, transformer, columns) tuples. The name is a string identifier used in logging and parameter access. The transformer is any scikit-learn estimator with fit and transform. The columns can be a list of column names (for DataFrames), integer indices (for arrays), or a boolean mask. All transformations are applied in parallel, and outputs are concatenated column-wise into a single matrix.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ordinal', OrdinalEncoder(
categories=[['high school', 'bachelor', 'master', 'PhD']]
), ['education']),
('ohe', OneHotEncoder(drop='first', sparse_output=False), ['city'])
])
# Output: scaled numeric + ordinal int + one-hot columns concatenatedFitting and Transforming Data
Like any scikit-learn transformer, ColumnTransformer has fit(), transform(), and fit_transform() methods. When you call fit(X_train), every sub-transformer is fitted on the training data simultaneously. When you call transform(X_test), each transformer applies its learnt parameters (e.g., scaler mean, encoder categories) to the test data. Each sub-transformer only touches its designated columns, so the fit statistics are clean and isolated.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split
import pandas as pd
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ohe', OneHotEncoder(sparse_output=False), ['city'])
])
ct.fit(X_train) # Learn stats from training only
X_train_t = ct.transform(X_train)
X_test_t = ct.transform(X_test) # Apply training stats to test
print('Transformed train shape:', X_train_t.shape)The remainder Parameter
By default, any column not explicitly listed in ColumnTransformer is dropped from the output. The remainder parameter controls this behaviour. Set remainder='passthrough' to keep unspecified columns unchanged (appended at the end of the output matrix). Set remainder=StandardScaler() to apply a transformer to all columns not mentioned. Using 'passthrough' is convenient when most columns are numeric and you only need to specify special treatment for a few categorical ones.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
# Only encode 'city'; pass all other columns through unchanged
ct = ColumnTransformer(
transformers=[
('ohe', OneHotEncoder(sparse_output=False), ['city'])
],
remainder='passthrough' # age, income, education appended as-is
)
X_transformed = ct.fit_transform(df)
print('Shape with passthrough:', X_transformed.shape)Getting Feature Names from ColumnTransformer
After transformation, the output matrix column count differs from the input because one-hot encoding adds columns. Call ct.get_feature_names_out() to retrieve the names of all output columns. Names are prefixed by the transformer name (e.g., ohe__city_NYC, scale__age). This is essential for interpreting feature importances, understanding SHAP values, or debugging when model performance is unexpectedly poor on specific features.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ohe', OneHotEncoder(sparse_output=False), ['city'])
])
ct.fit(df[['age', 'income', 'city']])
feature_names = ct.get_feature_names_out()
print('Output feature names:', feature_names)Column Specification Methods
ColumnTransformer supports multiple ways to specify which columns to transform: by name (string list — most readable), by index (integer list — for arrays without column names), or with a boolean selector using make_column_selector() to automatically detect numeric or categorical columns. The make_column_selector(dtype_include=np.number) pattern is especially powerful for DataFrames where you want to automatically scale all numeric columns without listing them individually.
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import numpy as np
# Automatically select numeric vs categorical columns
ct = ColumnTransformer([
('scale', StandardScaler(),
make_column_selector(dtype_include=np.number)),
('ohe', OneHotEncoder(sparse_output=False),
make_column_selector(dtype_include=object))
])
X_out = ct.fit_transform(df)
print('Auto-detected shape:', X_out.shape)Nesting ColumnTransformer in a Pipeline
The real power of ColumnTransformer emerges when it is nested inside a Pipeline. The ColumnTransformer handles preprocessing, and the downstream estimator handles learning. Wrapping both in a pipeline lets you pass raw DataFrames directly to fit() and predict(). When used with cross_val_score, the pipeline ensures that preprocessing is refitted on each training fold — preventing any test data from influencing transformation parameters.
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
preprocessor = ColumnTransformer([
('num', StandardScaler(), ['age', 'income']),
('cat', OneHotEncoder(handle_unknown='ignore'), ['city', 'education'])
])
pipeline = Pipeline([
('prep', preprocessor),
('clf', RandomForestClassifier(n_estimators=100, random_state=42))
])
scores = cross_val_score(pipeline, X, y, cv=5)
print('CV mean accuracy:', scores.mean().round(3))Grid Searching Pipeline Parameters
When your ColumnTransformer is inside a Pipeline, you can tune the hyperparameters of any step using GridSearchCV. Access nested parameters with double-underscore notation: prep__num__with_mean targets the with_mean parameter of the num transformer inside prep (the ColumnTransformer). Similarly, clf__n_estimators tunes the classifier. This makes end-to-end hyperparameter optimisation possible without manually re-running preprocessing.
from sklearn.model_selection import GridSearchCV
param_grid = {
'prep__num__with_mean': [True, False], # StandardScaler param
'clf__n_estimators': [50, 100, 200],
'clf__max_depth': [None, 5, 10]
}
grid = GridSearchCV(pipeline, param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
print('Best params:', grid.best_params_)
print('Best CV score:', grid.best_score_.round(3))Handling Mixed Imputation and Encoding
A complete preprocessing pipeline often chains imputation and encoding for the same column type. Scikit-learn supports nested pipelines: you can pass a Pipeline as the transformer inside a ColumnTransformer. For example, numeric columns might need imputation followed by scaling, while categorical columns need imputation followed by one-hot encoding. This eliminates any manual intermediate steps and keeps the entire transformation reproducible in a single fit call.
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
num_pipe = Pipeline([
('impute', SimpleImputer(strategy='median')),
('scale', StandardScaler())
])
cat_pipe = Pipeline([
('impute', SimpleImputer(strategy='most_frequent')),
('ohe', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])
preprocessor = ColumnTransformer([
('num', num_pipe, ['age', 'income']),
('cat', cat_pipe, ['city', 'education'])
])Inspecting Sub-Transformers
After fitting a ColumnTransformer, you can inspect each sub-transformer by name to debug or extract learned parameters. Use ct.named_transformers_ to access transformers by their string name. For a nested pipeline inside ColumnTransformer, chain attribute access: ct.named_transformers_['num']['scale'].mean_ retrieves the learned mean from the StandardScaler inside the numeric sub-pipeline. This introspection is invaluable for confirming that preprocessing ran correctly on the right columns.
# After fitting the preprocessor:
preprocessor.fit(X_train)
# Access the numeric sub-pipeline
num_pipeline = preprocessor.named_transformers_['num']
print('Imputer strategy:', num_pipeline['impute'].strategy)
print('Scaler mean:', num_pipeline['scale'].mean_)
# Access the OHE categories
cat_pipeline = preprocessor.named_transformers_['cat']
print('OHE categories:', cat_pipeline['ohe'].categories_)Persisting and Reloading a Fitted ColumnTransformer
A fitted ColumnTransformer (or the full pipeline containing it) stores all learned parameters: scaling means and standard deviations, encoder category lists, imputer fill values. Serialise it with joblib.dump() and reload it later without re-training. This is how preprocessing is deployed to production: the fitted transformer ensures that incoming data is processed identically to training data, so predictions are consistent and correct across sessions.
import joblib
# Save the fitted pipeline
joblib.dump(pipeline, 'model_pipeline.joblib')
# Load and use in production without re-training
loaded_pipeline = joblib.load('model_pipeline.joblib')
# New raw data in the same format as training data
X_new = pd.DataFrame({
'age': [30], 'income': [65000],
'city': ['NYC'], 'education': ['master']
})
prediction = loaded_pipeline.predict(X_new)
print('Prediction:', prediction)Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: how ColumnTransformer applies different transformations to different column subsets simultaneously, how to nest it inside a Pipeline for leak-free cross-validation, and how to compose imputation and encoding together using sub-pipelines. Next up we explore the K-Nearest Neighbors algorithm and how it classifies new data points using distance.
Frequently asked questions
Is the “Combining Steps with ColumnTransformer” lesson free?
Yes — the full text of “Combining Steps with ColumnTransformer” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Combining Steps with ColumnTransformer”?
Learners will apply different preprocessing steps to different column types simultaneously using ColumnTransformer, producing a single clean feature matrix. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Combining Steps with ColumnTransformer” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.