Interpolation and Advanced Imputation
Use interpolate() for smooth time-based filling and understand when mean vs. median imputation is appropriate.
Interpolation and Advanced Imputation is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
When Interpolation Beats fillna()
Forward fill and backward fill carry the nearest known value without considering the trend of the data. Interpolation estimates missing values by assuming a smooth transition between known values — for example, if temperature was 20°C on Monday and 30°C on Friday, interpolation estimates 25°C for Wednesday. This produces more realistic imputations for smoothly changing signals like sensor data, prices, or population counts.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'day': [1, 2, 3, 4, 5],
'temp': [20.0, np.nan, np.nan, np.nan, 30.0]
})
# Linear interpolation fills gaps smoothly
df['temp_interp'] = df['temp'].interpolate(method='linear')
print(df)
# day temp temp_interp
# 0 1 20.0 20.0
# 1 2 NaN 22.5
# 2 3 NaN 25.0
# 3 4 NaN 27.5
# 4 5 30.0 30.0interpolate() Methods
Pandas interpolate() supports multiple methods. The most common are 'linear' (equally spaced between known values), 'time' (accounts for unequal time gaps when a DatetimeIndex is set), 'polynomial' (fits a polynomial curve, requires an order argument), and 'spline' (smooth piecewise polynomial). Linear is the safest default; higher-order methods can overfit small gaps.
import pandas as pd
import numpy as np
s = pd.Series([0, np.nan, np.nan, 8.0])
print('linear:', s.interpolate('linear').tolist())
# [0.0, 2.6666..., 5.3333..., 8.0]
print('polynomial(2):', s.interpolate('polynomial', order=2).tolist())
# [0.0, 2.222..., 5.111..., 8.0]time Interpolation with DatetimeIndex
When your Series has a DatetimeIndex with unequal time gaps (e.g., weekdays only, missing weekends), method='time' interpolates proportionally to the actual elapsed time, not just by position. This is more accurate than linear interpolation, which treats each row as equally spaced regardless of the calendar gap.
import pandas as pd
import numpy as np
# Unequal gaps: Jan 1, Jan 3, Jan 10
idx = pd.to_datetime(['2024-01-01', '2024-01-03', '2024-01-10'])
s = pd.Series([100.0, np.nan, 170.0], index=idx)
# linear treats gaps as equal (position-based)
print(s.interpolate('linear').tolist()) # [100.0, 135.0, 170.0]
# time accounts for actual day count
# Jan 1 to Jan 3 = 2 days, Jan 1 to Jan 10 = 9 days
# fraction: 2/9 of the way from 100 to 170
print(s.interpolate('time').tolist()) # [100.0, 115.55..., 170.0]Limiting Interpolation Extent
Like ffill(), interpolate() accepts a limit parameter to restrict how many consecutive NaN positions are filled. Remaining NaN beyond the limit stay missing. This prevents interpolation from running across very long gaps where the true value could be anything — long gaps should be flagged for manual review.
import pandas as pd
import numpy as np
s = pd.Series([1.0, np.nan, np.nan, np.nan, np.nan, 10.0])
# Only fill the first 2 NaN positions
filled = s.interpolate('linear', limit=2)
print(filled.tolist())
# [1.0, 2.8, 4.6, NaN, NaN, 10.0]Choosing Mean vs. Median Imputation
Mean and median imputation are simple but have important trade-offs. The mean is sensitive to outliers — a few very large values pull it up, making it a poor fill for a right-skewed distribution like income or house prices. The median is robust to outliers and is the better choice for skewed columns. Use the mean only when your data is roughly symmetric and free of extreme outliers.
import pandas as pd
import numpy as np
# Skewed income data with an outlier
income = pd.Series([30000, 35000, 32000, 1_000_000, np.nan])
print('Mean: ', income.mean()) # ~274000 — pulled by outlier
print('Median:', income.median()) # ~33500 — robust choice
# Prefer median for skewed data
filled = income.fillna(income.median())
print(filled.tolist())
# [30000, 35000, 32000, 1000000, 33500.0]KNN Imputation Concept
K-Nearest Neighbour (KNN) imputation replaces a missing value with the mean (or weighted mean) of the k most similar rows, measured by the distance across non-missing features. This is a multivariate strategy — it uses information from other columns to inform the fill, which is more accurate than single-column mean imputation when features are correlated.
Scikit-learn's KNNImputer integrates directly with NumPy arrays and Pandas DataFrames.
import pandas as pd
import numpy as np
from sklearn.impute import KNNImputer
df = pd.DataFrame({
'age': [25, 35, np.nan, 45],
'income': [50000, 70000, 60000, np.nan]
})
imputer = KNNImputer(n_neighbors=2)
df_imputed = pd.DataFrame(
imputer.fit_transform(df),
columns=df.columns
)
print(df_imputed.round(1))
# age income
# 0 25.0 50000.0
# 1 35.0 70000.0
# 2 30.0 60000.0 <- filled from neighbours
# 3 45.0 65000.0 <- filled from neighboursMultiple Imputation with IterativeImputer
Multiple imputation is the gold standard for handling missing data in statistical research. Scikit-learn's IterativeImputer implements a MICE (Multiple Imputation by Chained Equations) approach: it models each column with missing values as a function of the other columns, iterating the imputation until values converge. This captures cross-feature relationships better than single-column strategies.
import pandas as pd
import numpy as np
from sklearn.experimental import enable_iterative_imputer # noqa
from sklearn.impute import IterativeImputer
df = pd.DataFrame({
'x1': [1, 2, np.nan, 4, 5],
'x2': [2, np.nan, 6, 8, 10],
'y': [3, 5, 7, np.nan, 11]
})
imp = IterativeImputer(max_iter=10, random_state=0)
df_filled = pd.DataFrame(imp.fit_transform(df), columns=df.columns)
print(df_filled.round(2))Indicator Columns for Missing Data
Rather than hiding the fact that a value was imputed, it is good practice to add a binary indicator column that flags which rows had missing values before imputation. This lets downstream models learn from the missingness pattern itself — sometimes whether a value is missing is as predictive as the value itself.
import pandas as pd
import numpy as np
df = pd.DataFrame({'salary': [50000, np.nan, 70000, np.nan, 90000]})
# Add indicator before filling
df['salary_was_missing'] = df['salary'].isna().astype(int)
# Then fill
df['salary'] = df['salary'].fillna(df['salary'].median())
print(df)
# salary salary_was_missing
# 0 50000.0 0
# 1 70000.0 1
# 2 70000.0 0
# 3 70000.0 1
# 4 90000.0 0Imputation and Data Leakage
A critical rule in machine learning pipelines: compute imputation statistics (mean, median, mode) on the training set only, then apply the same values to the test set. Computing statistics on the full dataset before splitting causes data leakage — the model indirectly sees test data during training, inflating performance estimates. Always fit imputers on training data and transform both train and test.
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
df = pd.DataFrame({'feature': [1, 2, np.nan, 4, np.nan, 6, 7, 8]})
train, test = train_test_split(df, test_size=0.25, random_state=0)
# Fit ONLY on training data
imp = SimpleImputer(strategy='mean')
train['feature'] = imp.fit_transform(train[['feature']])
# Transform test using training statistics
test['feature'] = imp.transform(test[['feature']])
print('Train mean used:', imp.statistics_[0])Comparing Imputation Strategies Visually
After applying multiple imputation methods, compare their effect by plotting the distribution of the original and imputed columns side by side. A good imputation should preserve the shape (mean, variance, skewness) of the original distribution as closely as possible. Mean imputation narrows the distribution; KNN and MICE tend to preserve it better.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
original = np.random.exponential(scale=5, size=200)
with_nan = original.copy()
with_nan[np.random.choice(200, 40, replace=False)] = np.nan
s = pd.Series(with_nan)
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
s.dropna().hist(ax=axes[0], bins=20, title='Original (no NaN)')
s.fillna(s.mean()).hist(ax=axes[1], bins=20, title='Mean-imputed')
plt.tight_layout()
plt.savefig('imputation_comparison.png')
print('Saved comparison chart')Choosing the Right Imputation Strategy
There is no universally best imputation strategy — the right choice depends on context. Use mean/median for simple univariate cases with low missingness. Use ffill/bfill for time series with stable trends. Use KNN or IterativeImputer when features are correlated and you want to leverage relationships. Use domain constants (e.g., 0 for absent, -1 for unknown) when the missing value has a clear business meaning. Always document your choice.
# Decision guide (no runnable code)
#
# Missing < 5% AND data is MCAR? -> Mean/median fill is fine
# Time series with stable trend? -> ffill() with limit
# Features are correlated? -> KNNImputer or IterativeImputer
# Categorical column? -> Mode fill or 'Unknown' sentinel
# Going into ML model? -> Add indicator column + fill
# Research / publication quality? -> Multiple imputation (MICE)
print('Strategy selected based on context')Quick Check
Test your understanding of interpolation and advanced imputation.
Lesson Recap
In this lesson you learned: interpolate() estimates missing values by assuming a smooth trend between known values, method='time' handles unequal DatetimeIndex gaps, and advanced strategies like KNNImputer and IterativeImputer use other columns to inform fills. Always add indicator columns before imputation and fit statistics on the training set only to avoid data leakage. Next up we inspect and convert DataFrame column data types.
Frequently asked questions
Is the “Interpolation and Advanced Imputation” lesson free?
Yes — the full text of “Interpolation and Advanced Imputation” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Interpolation and Advanced Imputation”?
Use interpolate() for smooth time-based filling and understand when mean vs. median imputation is appropriate. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Interpolation and Advanced Imputation” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Detecting Missing Values
- Dropping Missing Values
- Filling Missing Values
- Interpolation and Advanced Imputation