ColumnTransformerによる処理の組み合わせ
ColumnTransformerを使用して異なる列型に異なる前処理を同時に適用し、単一のクリーンな特徴量行列を生成します。
「ColumnTransformerによる処理の組み合わせ」はCoddyKit上の無料Machine Learning Academyレッスンです。 これはレッスン4/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはMachine Learning Academy学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Machine Learning Academyコースには全4レッスンが含まれています。
このレッスンの一部はまだ翻訳されておらず、英語で表示されています。
The Problem ColumnTransformer Solves
Real datasets contain a mix of column types: some are numeric, some are ordinal, and some are nominal categorical. Applying the same transformer to all columns would be incorrect — you cannot StandardScale a one-hot column or OrdinalEncode a continuous number. Before ColumnTransformer, data scientists had to manually slice DataFrames, apply different transformers, and re-concatenate — a fragile, leakage-prone process. ColumnTransformer solves this by applying different transformations to different column subsets simultaneously inside a single, pipeline-compatible object.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'age': [25, 35, 45, 28],
'income': [40000, 80000, 120000, 55000],
'education': ['bachelor', 'master', 'PhD', 'high school'],
'city': ['NYC', 'LA', 'NYC', 'Chicago']
})
# Need: StandardScaler for age+income
# OrdinalEncoder for education
# OneHotEncoder for city
# ColumnTransformer applies all three at onceColumnTransformer Syntax and Structure
A ColumnTransformer is constructed as a list of (name, transformer, columns) tuples. The name is a string identifier used in logging and parameter access. The transformer is any scikit-learn estimator with fit and transform. The columns can be a list of column names (for DataFrames), integer indices (for arrays), or a boolean mask. All transformations are applied in parallel, and outputs are concatenated column-wise into a single matrix.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ordinal', OrdinalEncoder(
categories=[['high school', 'bachelor', 'master', 'PhD']]
), ['education']),
('ohe', OneHotEncoder(drop='first', sparse_output=False), ['city'])
])
# Output: scaled numeric + ordinal int + one-hot columns concatenatedFitting and Transforming Data
Like any scikit-learn transformer, ColumnTransformer has fit(), transform(), and fit_transform() methods. When you call fit(X_train), every sub-transformer is fitted on the training data simultaneously. When you call transform(X_test), each transformer applies its learnt parameters (e.g., scaler mean, encoder categories) to the test data. Each sub-transformer only touches its designated columns, so the fit statistics are clean and isolated.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split
import pandas as pd
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ohe', OneHotEncoder(sparse_output=False), ['city'])
])
ct.fit(X_train) # Learn stats from training only
X_train_t = ct.transform(X_train)
X_test_t = ct.transform(X_test) # Apply training stats to test
print('Transformed train shape:', X_train_t.shape)The remainder Parameter
By default, any column not explicitly listed in ColumnTransformer is dropped from the output. The remainder parameter controls this behaviour. Set remainder='passthrough' to keep unspecified columns unchanged (appended at the end of the output matrix). Set remainder=StandardScaler() to apply a transformer to all columns not mentioned. Using 'passthrough' is convenient when most columns are numeric and you only need to specify special treatment for a few categorical ones.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
# Only encode 'city'; pass all other columns through unchanged
ct = ColumnTransformer(
transformers=[
('ohe', OneHotEncoder(sparse_output=False), ['city'])
],
remainder='passthrough' # age, income, education appended as-is
)
X_transformed = ct.fit_transform(df)
print('Shape with passthrough:', X_transformed.shape)Getting Feature Names from ColumnTransformer
After transformation, the output matrix column count differs from the input because one-hot encoding adds columns. Call ct.get_feature_names_out() to retrieve the names of all output columns. Names are prefixed by the transformer name (e.g., ohe__city_NYC, scale__age). This is essential for interpreting feature importances, understanding SHAP values, or debugging when model performance is unexpectedly poor on specific features.
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
ct = ColumnTransformer([
('scale', StandardScaler(), ['age', 'income']),
('ohe', OneHotEncoder(sparse_output=False), ['city'])
])
ct.fit(df[['age', 'income', 'city']])
feature_names = ct.get_feature_names_out()
print('Output feature names:', feature_names)Column Specification Methods
ColumnTransformer supports multiple ways to specify which columns to transform: by name (string list — most readable), by index (integer list — for arrays without column names), or with a boolean selector using make_column_selector() to automatically detect numeric or categorical columns. The make_column_selector(dtype_include=np.number) pattern is especially powerful for DataFrames where you want to automatically scale all numeric columns without listing them individually.
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import numpy as np
# Automatically select numeric vs categorical columns
ct = ColumnTransformer([
('scale', StandardScaler(),
make_column_selector(dtype_include=np.number)),
('ohe', OneHotEncoder(sparse_output=False),
make_column_selector(dtype_include=object))
])
X_out = ct.fit_transform(df)
print('Auto-detected shape:', X_out.shape)Nesting ColumnTransformer in a Pipeline
The real power of ColumnTransformer emerges when it is nested inside a Pipeline. The ColumnTransformer handles preprocessing, and the downstream estimator handles learning. Wrapping both in a pipeline lets you pass raw DataFrames directly to fit() and predict(). When used with cross_val_score, the pipeline ensures that preprocessing is refitted on each training fold — preventing any test data from influencing transformation parameters.
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
preprocessor = ColumnTransformer([
('num', StandardScaler(), ['age', 'income']),
('cat', OneHotEncoder(handle_unknown='ignore'), ['city', 'education'])
])
pipeline = Pipeline([
('prep', preprocessor),
('clf', RandomForestClassifier(n_estimators=100, random_state=42))
])
scores = cross_val_score(pipeline, X, y, cv=5)
print('CV mean accuracy:', scores.mean().round(3))Grid Searching Pipeline Parameters
When your ColumnTransformer is inside a Pipeline, you can tune the hyperparameters of any step using GridSearchCV. Access nested parameters with double-underscore notation: prep__num__with_mean targets the with_mean parameter of the num transformer inside prep (the ColumnTransformer). Similarly, clf__n_estimators tunes the classifier. This makes end-to-end hyperparameter optimisation possible without manually re-running preprocessing.
from sklearn.model_selection import GridSearchCV
param_grid = {
'prep__num__with_mean': [True, False], # StandardScaler param
'clf__n_estimators': [50, 100, 200],
'clf__max_depth': [None, 5, 10]
}
grid = GridSearchCV(pipeline, param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
print('Best params:', grid.best_params_)
print('Best CV score:', grid.best_score_.round(3))Handling Mixed Imputation and Encoding
A complete preprocessing pipeline often chains imputation and encoding for the same column type. Scikit-learn supports nested pipelines: you can pass a Pipeline as the transformer inside a ColumnTransformer. For example, numeric columns might need imputation followed by scaling, while categorical columns need imputation followed by one-hot encoding. This eliminates any manual intermediate steps and keeps the entire transformation reproducible in a single fit call.
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
num_pipe = Pipeline([
('impute', SimpleImputer(strategy='median')),
('scale', StandardScaler())
])
cat_pipe = Pipeline([
('impute', SimpleImputer(strategy='most_frequent')),
('ohe', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])
preprocessor = ColumnTransformer([
('num', num_pipe, ['age', 'income']),
('cat', cat_pipe, ['city', 'education'])
])Inspecting Sub-Transformers
After fitting a ColumnTransformer, you can inspect each sub-transformer by name to debug or extract learned parameters. Use ct.named_transformers_ to access transformers by their string name. For a nested pipeline inside ColumnTransformer, chain attribute access: ct.named_transformers_['num']['scale'].mean_ retrieves the learned mean from the StandardScaler inside the numeric sub-pipeline. This introspection is invaluable for confirming that preprocessing ran correctly on the right columns.
# After fitting the preprocessor:
preprocessor.fit(X_train)
# Access the numeric sub-pipeline
num_pipeline = preprocessor.named_transformers_['num']
print('Imputer strategy:', num_pipeline['impute'].strategy)
print('Scaler mean:', num_pipeline['scale'].mean_)
# Access the OHE categories
cat_pipeline = preprocessor.named_transformers_['cat']
print('OHE categories:', cat_pipeline['ohe'].categories_)Persisting and Reloading a Fitted ColumnTransformer
A fitted ColumnTransformer (or the full pipeline containing it) stores all learned parameters: scaling means and standard deviations, encoder category lists, imputer fill values. Serialise it with joblib.dump() and reload it later without re-training. This is how preprocessing is deployed to production: the fitted transformer ensures that incoming data is processed identically to training data, so predictions are consistent and correct across sessions.
import joblib
# Save the fitted pipeline
joblib.dump(pipeline, 'model_pipeline.joblib')
# Load and use in production without re-training
loaded_pipeline = joblib.load('model_pipeline.joblib')
# New raw data in the same format as training data
X_new = pd.DataFrame({
'age': [30], 'income': [65000],
'city': ['NYC'], 'education': ['master']
})
prediction = loaded_pipeline.predict(X_new)
print('Prediction:', prediction)Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: how ColumnTransformer applies different transformations to different column subsets simultaneously, how to nest it inside a Pipeline for leak-free cross-validation, and how to compose imputation and encoding together using sub-pipelines. Next up we explore the K-Nearest Neighbors algorithm and how it classifies new data points using distance.
よくある質問
「ColumnTransformerによる処理の組み合わせ」レッスンは無料ですか？
はい。「ColumnTransformerによる処理の組み合わせ」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Machine Learning Academyコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Machine Learning Academyコースには全4レッスンが含まれています。
「ColumnTransformerによる処理の組み合わせ」で何を学びますか？
ColumnTransformerを使用して異なる列型に異なる前処理を同時に適用し、単一のクリーンな特徴量行列を生成します。 ブラウザで直接実行するハンズオンコードでMachine Learning Academyを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。
Machine Learning Academyを始めるのに経験は必要ですか？
事前経験は必要ありません。CoddyKitのMachine Learning Academyは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン4/4です。
「ColumnTransformerによる処理の組み合わせ」レッスンにはどのくらい時間がかかりますか？
ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。
このMachine Learning Academyレッスンでコードを書いて実行できますか？
はい。すべてのMachine Learning Academyレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。
このコースのすべてのレッスン
- 欠損値への対処：削除、補完、フラグ付け
- 特徴量のスケーリング：StandardScalerとMinMaxScaler
- カテゴリ変数のエンコード：OrdinalEncoderとOneHotEncoder
- ColumnTransformerによる処理の組み合わせ