Handling Missing Values: Drop, Impute, and Flag

Missing values — represented as NaN, None, or empty cells — are one of the most common data quality issues in real-world datasets. If left unhandled, they cause errors in scikit-learn estimators, which expect fully numeric matrices. Every ML pipeline must address missing values before training. There are three main strategies: drop rows/columns, impute (fill in estimated values), or flag the missingness as its own feature.

Before deciding how to handle missing values, you need to quantify the problem. Pandas provides df.isnull() to create a boolean mask and df.isnull().sum() to count nulls per column. The df.info() method also shows non-null counts. Always inspect the fraction of missing values — if more than 50% of a column is missing, that column may need to be dropped entirely rather than imputed.

df.dropna() removes rows containing any null by default. Pass axis=1 to drop columns instead. The thresh parameter lets you keep rows that have at least N non-null values. Dropping is safe when missingness is completely random (MCAR) and the fraction is tiny — less than 1-5% of rows. However, dropping too aggressively causes information loss and can introduce bias if the data is not missing at random.

Imputation fills missing values with estimated substitutes. Mean imputation replaces nulls with the column mean, which works well for symmetric, normally distributed data. Median imputation is more robust to outliers and is preferred for skewed distributions. Both are simple and fast but reduce variance artificially since all imputed values are identical. Use SimpleImputer from scikit-learn for pipeline-compatible imputation.

For categorical columns, using mean or median makes no sense. Instead, use strategy='most_frequent' in SimpleImputer, which fills missing values with the mode (most common category). This preserves the distribution shape better than random assignment. Always fit the imputer only on training data and use the fitted imputer to transform both train and test sets — this prevents data leakage from the test set influencing the imputed values.

Sometimes the best imputed value is a domain-specific constant, not a statistic. For example, missing number of previous purchases likely means zero. Use strategy='constant' with fill_value=0 in SimpleImputer. For more complex logic — like filling missing city with 'Unknown' — pass a string constant. Constant imputation is interpretable and safe when absence genuinely implies a specific value rather than data collection failure.

More sophisticated imputers model the relationship between features to produce better estimates. KNNImputer fills a missing value by looking at the k nearest complete rows and averaging their values — preserving local data structure. IterativeImputer (experimental) fits a separate regression model for each feature with nulls, using all other features as predictors. Both are more accurate than mean imputation but are slower and can overfit on small datasets.

Sometimes the fact that a value is missing is itself informative. For instance, a missing salary field might indicate unemployment. Adding a binary indicator column that marks whether each value was originally null allows the model to learn from the missingness pattern. MissingIndicator in scikit-learn creates these flag columns. Use it together with imputation in a FeatureUnion or ColumnTransformer so the model sees both the imputed value and the flag.

A critical rule: always fit the imputer on training data only, then use the fitted imputer to transform both train and test sets. Fitting on the full dataset leaks test statistics into training, giving optimistic evaluation results. This applies to every statistic computed during preprocessing — mean, median, mode, or nearest-neighbor distances. Use scikit-learn Pipeline to enforce this automatically: the pipeline fits preprocessing steps only on the training fold during cross-validation.

Choosing the right strategy depends on three factors: (1) fraction missing — below 5% dropping rows is acceptable; above 50% consider dropping the column; (2) mechanism — MCAR (completely random) favours dropping; MAR (random given other features) favours imputation; MNAR (not at random) favours flagging; (3) downstream model — tree-based models handle missing data natively (set allow_nan=True in XGBoost/LightGBM), so imputation may be optional. Always cross-validate to confirm your strategy improves model performance.

Here is a complete example combining imputation into a scikit-learn pipeline. ColumnTransformer applies SimpleImputer with strategy='median' to numeric columns and strategy='most_frequent' to categorical columns simultaneously. The result flows into a classifier, and the whole pipeline is safely cross-validated so imputation statistics never leak between folds. This pattern is the production-ready approach to missing data handling in any real ML project.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: how to detect missing values using isnull().sum() and info(), three main strategies — drop, impute, and flag — and when each is appropriate, and how to use SimpleImputer, KNNImputer, and MissingIndicator within a pipeline to prevent leakage. Next up we explore feature scaling with StandardScaler and MinMaxScaler.