使用 ColumnTransformer 组合步骤

Real datasets contain a mix of column types: some are numeric, some are ordinal, and some are nominal categorical. Applying the same transformer to all columns would be incorrect — you cannot StandardScale a one-hot column or OrdinalEncode a continuous number. Before ColumnTransformer, data scientists had to manually slice DataFrames, apply different transformers, and re-concatenate — a fragile, leakage-prone process. ColumnTransformer solves this by applying different transformations to different column subsets simultaneously inside a single, pipeline-compatible object.

A ColumnTransformer is constructed as a list of (name, transformer, columns) tuples. The name is a string identifier used in logging and parameter access. The transformer is any scikit-learn estimator with fit and transform. The columns can be a list of column names (for DataFrames), integer indices (for arrays), or a boolean mask. All transformations are applied in parallel, and outputs are concatenated column-wise into a single matrix.

Like any scikit-learn transformer, ColumnTransformer has fit(), transform(), and fit_transform() methods. When you call fit(X_train), every sub-transformer is fitted on the training data simultaneously. When you call transform(X_test), each transformer applies its learnt parameters (e.g., scaler mean, encoder categories) to the test data. Each sub-transformer only touches its designated columns, so the fit statistics are clean and isolated.

By default, any column not explicitly listed in ColumnTransformer is dropped from the output. The remainder parameter controls this behaviour. Set remainder='passthrough' to keep unspecified columns unchanged (appended at the end of the output matrix). Set remainder=StandardScaler() to apply a transformer to all columns not mentioned. Using 'passthrough' is convenient when most columns are numeric and you only need to specify special treatment for a few categorical ones.

After transformation, the output matrix column count differs from the input because one-hot encoding adds columns. Call ct.get_feature_names_out() to retrieve the names of all output columns. Names are prefixed by the transformer name (e.g., ohe__city_NYC, scale__age). This is essential for interpreting feature importances, understanding SHAP values, or debugging when model performance is unexpectedly poor on specific features.

ColumnTransformer supports multiple ways to specify which columns to transform: by name (string list — most readable), by index (integer list — for arrays without column names), or with a boolean selector using make_column_selector() to automatically detect numeric or categorical columns. The make_column_selector(dtype_include=np.number) pattern is especially powerful for DataFrames where you want to automatically scale all numeric columns without listing them individually.

The real power of ColumnTransformer emerges when it is nested inside a Pipeline. The ColumnTransformer handles preprocessing, and the downstream estimator handles learning. Wrapping both in a pipeline lets you pass raw DataFrames directly to fit() and predict(). When used with cross_val_score, the pipeline ensures that preprocessing is refitted on each training fold — preventing any test data from influencing transformation parameters.

When your ColumnTransformer is inside a Pipeline, you can tune the hyperparameters of any step using GridSearchCV. Access nested parameters with double-underscore notation: prep__num__with_mean targets the with_mean parameter of the num transformer inside prep (the ColumnTransformer). Similarly, clf__n_estimators tunes the classifier. This makes end-to-end hyperparameter optimisation possible without manually re-running preprocessing.

A complete preprocessing pipeline often chains imputation and encoding for the same column type. Scikit-learn supports nested pipelines: you can pass a Pipeline as the transformer inside a ColumnTransformer. For example, numeric columns might need imputation followed by scaling, while categorical columns need imputation followed by one-hot encoding. This eliminates any manual intermediate steps and keeps the entire transformation reproducible in a single fit call.

After fitting a ColumnTransformer, you can inspect each sub-transformer by name to debug or extract learned parameters. Use ct.named_transformers_ to access transformers by their string name. For a nested pipeline inside ColumnTransformer, chain attribute access: ct.named_transformers_['num']['scale'].mean_ retrieves the learned mean from the StandardScaler inside the numeric sub-pipeline. This introspection is invaluable for confirming that preprocessing ran correctly on the right columns.

A fitted ColumnTransformer (or the full pipeline containing it) stores all learned parameters: scaling means and standard deviations, encoder category lists, imputer fill values. Serialise it with joblib.dump() and reload it later without re-training. This is how preprocessing is deployed to production: the fitted transformer ensures that incoming data is processed identically to training data, so predictions are consistent and correct across sessions.

In this lesson you learned: how ColumnTransformer applies different transformations to different column subsets simultaneously, how to nest it inside a Pipeline for leak-free cross-validation, and how to compose imputation and encoding together using sub-pipelines. Next up we explore the K-Nearest Neighbors algorithm and how it classifies new data points using distance.