Encoding Categorical Variables: OrdinalEncoder and OneHotEncoder

Machine learning models operate on numbers, not strings. A column containing values like 'red', 'green', 'blue' cannot be fed directly into a scikit-learn estimator. You must convert categories to numeric representations before training. The choice of encoding method matters greatly: a poor encoding can introduce spurious ordinal relationships or explode dimensionality. This lesson covers the two most important encoders: OrdinalEncoder for ordered categories and OneHotEncoder for unordered ones.

Not all categorical variables are equal. Ordinal categories have a meaningful order: small < medium < large, or bad < fair < good < excellent. Nominal categories have no intrinsic order: red, blue, green are just labels. Choosing the wrong encoding creates false relationships — for example, integer-encoding unordered colors as 0, 1, 2 tells the model that blue (1) is somehow between red (0) and green (2), which is meaningless. Always identify whether a variable is ordinal or nominal before encoding.

OrdinalEncoder maps each category to an integer based on a specified order. You provide the categories parameter as a list of lists, where the position determines the integer assigned. If you do not specify order, scikit-learn assigns integers alphabetically — which may or may not align with true order. Always specify the category order explicitly for ordinal variables to ensure the model captures the correct relationship between values.

OneHotEncoder creates one binary column per category. For a color feature with values red, green, blue, it produces three columns: color_red, color_green, color_blue, where exactly one is 1 and the rest are 0 for each row. This avoids any implied ordering. The trade-off is dimensionality: a feature with 100 unique values creates 100 new columns. By default, scikit-learn's OneHotEncoder drops one category (dummy encoding) to avoid perfect multicollinearity in linear models.

When using OneHotEncoder with linear models, keeping all K columns for a K-category variable introduces perfect multicollinearity — the columns sum to 1, so any one column is a linear combination of the others. This makes the design matrix singular (non-invertible). Setting drop='first' drops the first category, leaving K-1 columns that fully describe the variable without redundancy. For tree-based models, keeping all K columns is harmless and sometimes slightly better, so you can use drop=None.

A real-world challenge: at inference time, a category may appear that was not in the training data. By default, OrdinalEncoder and OneHotEncoder raise an error on unknown categories. Set handle_unknown='ignore' in OneHotEncoder to produce all-zero rows for unknown values (the model predicts without that feature). For OrdinalEncoder, set handle_unknown='use_encoded_value' with unknown_value=-1 to flag unknowns explicitly.

When a categorical feature has hundreds or thousands of unique values (e.g., zip codes, product SKUs), one-hot encoding creates an unmanageably large matrix. Target encoding replaces each category with the mean of the target variable for that category. This compresses high-cardinality features to a single column. The risk is target leakage: if you encode using the mean computed on the same rows you train on, the model memorises patterns rather than generalising. Always use cross-fold target encoding to prevent leakage.

Binary encoding is a middle ground between ordinal and one-hot encoding. Each integer code is converted to its binary representation, producing log2(K) columns instead of K columns. For example, 8 categories need only 3 binary columns instead of 8 one-hot columns. This reduces dimensionality significantly for features with 10–100 categories. The category_encoders library provides BinaryEncoder that handles this transformation with a scikit-learn compatible API.

Real datasets have a mix of numeric, ordinal, and nominal columns. ColumnTransformer applies different transformations to different columns in parallel and concatenates the results. Specify ordinal columns with OrdinalEncoder, nominal columns with OneHotEncoder, and numeric columns with StandardScaler. The output is a single clean numeric matrix ready for any scikit-learn estimator. This is the standard production pattern for mixed-type datasets.

After encoding, your feature matrix has more columns than the original DataFrame, and scikit-learn transformers track the new names. Call get_feature_names_out() on ColumnTransformer or OneHotEncoder to retrieve the generated feature names. This is essential for model interpretability: knowing which one-hot column corresponds to which original category lets you correctly interpret feature importances from Random Forests or SHAP values from gradient boosting models.

Here is a complete pipeline with mixed encoding. ColumnTransformer handles three column types: numerical features are scaled, an ordinal feature is integer-encoded with explicit order, and nominal features become one-hot columns. The pipeline is then cross-validated — encoding is refitted within each CV fold to prevent leakage. This pattern works for any dataset and is ready for deployment once trained on the full dataset.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: the difference between ordinal and nominal categories and why it determines encoding choice, OrdinalEncoder for ordered variables and OneHotEncoder for nominal variables, and how to handle unknown categories and combine encoders in a ColumnTransformer pipeline. Next up we explore combining all preprocessing steps with ColumnTransformer.