Interpolation et imputation avancée
Utilisez interpolate() pour un remplissage lissé fondé sur le temps et comprenez quand l’imputation par la moyenne ou la médiane est appropriée.
Interpolation et imputation avancée est une leçon Pandas & NumPy Academy gratuite sur CoddyKit. Ceci est la leçon 4 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Pandas & NumPy Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Pandas & NumPy Academy comprend 4 leçons au total.
Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.
When Interpolation Beats fillna()
Forward fill and backward fill carry the nearest known value without considering the trend of the data. Interpolation estimates missing values by assuming a smooth transition between known values — for example, if temperature was 20°C on Monday and 30°C on Friday, interpolation estimates 25°C for Wednesday. This produces more realistic imputations for smoothly changing signals like sensor data, prices, or population counts.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'day': [1, 2, 3, 4, 5],
'temp': [20.0, np.nan, np.nan, np.nan, 30.0]
})
# Linear interpolation fills gaps smoothly
df['temp_interp'] = df['temp'].interpolate(method='linear')
print(df)
# day temp temp_interp
# 0 1 20.0 20.0
# 1 2 NaN 22.5
# 2 3 NaN 25.0
# 3 4 NaN 27.5
# 4 5 30.0 30.0interpolate() Methods
Pandas interpolate() supports multiple methods. The most common are 'linear' (equally spaced between known values), 'time' (accounts for unequal time gaps when a DatetimeIndex is set), 'polynomial' (fits a polynomial curve, requires an order argument), and 'spline' (smooth piecewise polynomial). Linear is the safest default; higher-order methods can overfit small gaps.
import pandas as pd
import numpy as np
s = pd.Series([0, np.nan, np.nan, 8.0])
print('linear:', s.interpolate('linear').tolist())
# [0.0, 2.6666..., 5.3333..., 8.0]
print('polynomial(2):', s.interpolate('polynomial', order=2).tolist())
# [0.0, 2.222..., 5.111..., 8.0]time Interpolation with DatetimeIndex
When your Series has a DatetimeIndex with unequal time gaps (e.g., weekdays only, missing weekends), method='time' interpolates proportionally to the actual elapsed time, not just by position. This is more accurate than linear interpolation, which treats each row as equally spaced regardless of the calendar gap.
import pandas as pd
import numpy as np
# Unequal gaps: Jan 1, Jan 3, Jan 10
idx = pd.to_datetime(['2024-01-01', '2024-01-03', '2024-01-10'])
s = pd.Series([100.0, np.nan, 170.0], index=idx)
# linear treats gaps as equal (position-based)
print(s.interpolate('linear').tolist()) # [100.0, 135.0, 170.0]
# time accounts for actual day count
# Jan 1 to Jan 3 = 2 days, Jan 1 to Jan 10 = 9 days
# fraction: 2/9 of the way from 100 to 170
print(s.interpolate('time').tolist()) # [100.0, 115.55..., 170.0]Limiting Interpolation Extent
Like ffill(), interpolate() accepts a limit parameter to restrict how many consecutive NaN positions are filled. Remaining NaN beyond the limit stay missing. This prevents interpolation from running across very long gaps where the true value could be anything — long gaps should be flagged for manual review.
import pandas as pd
import numpy as np
s = pd.Series([1.0, np.nan, np.nan, np.nan, np.nan, 10.0])
# Only fill the first 2 NaN positions
filled = s.interpolate('linear', limit=2)
print(filled.tolist())
# [1.0, 2.8, 4.6, NaN, NaN, 10.0]Choosing Mean vs. Median Imputation
Mean and median imputation are simple but have important trade-offs. The mean is sensitive to outliers — a few very large values pull it up, making it a poor fill for a right-skewed distribution like income or house prices. The median is robust to outliers and is the better choice for skewed columns. Use the mean only when your data is roughly symmetric and free of extreme outliers.
import pandas as pd
import numpy as np
# Skewed income data with an outlier
income = pd.Series([30000, 35000, 32000, 1_000_000, np.nan])
print('Mean: ', income.mean()) # ~274000 — pulled by outlier
print('Median:', income.median()) # ~33500 — robust choice
# Prefer median for skewed data
filled = income.fillna(income.median())
print(filled.tolist())
# [30000, 35000, 32000, 1000000, 33500.0]KNN Imputation Concept
K-Nearest Neighbour (KNN) imputation replaces a missing value with the mean (or weighted mean) of the k most similar rows, measured by the distance across non-missing features. This is a multivariate strategy — it uses information from other columns to inform the fill, which is more accurate than single-column mean imputation when features are correlated.
Scikit-learn's KNNImputer integrates directly with NumPy arrays and Pandas DataFrames.
import pandas as pd
import numpy as np
from sklearn.impute import KNNImputer
df = pd.DataFrame({
'age': [25, 35, np.nan, 45],
'income': [50000, 70000, 60000, np.nan]
})
imputer = KNNImputer(n_neighbors=2)
df_imputed = pd.DataFrame(
imputer.fit_transform(df),
columns=df.columns
)
print(df_imputed.round(1))
# age income
# 0 25.0 50000.0
# 1 35.0 70000.0
# 2 30.0 60000.0 <- filled from neighbours
# 3 45.0 65000.0 <- filled from neighboursMultiple Imputation with IterativeImputer
Multiple imputation is the gold standard for handling missing data in statistical research. Scikit-learn's IterativeImputer implements a MICE (Multiple Imputation by Chained Equations) approach: it models each column with missing values as a function of the other columns, iterating the imputation until values converge. This captures cross-feature relationships better than single-column strategies.
import pandas as pd
import numpy as np
from sklearn.experimental import enable_iterative_imputer # noqa
from sklearn.impute import IterativeImputer
df = pd.DataFrame({
'x1': [1, 2, np.nan, 4, 5],
'x2': [2, np.nan, 6, 8, 10],
'y': [3, 5, 7, np.nan, 11]
})
imp = IterativeImputer(max_iter=10, random_state=0)
df_filled = pd.DataFrame(imp.fit_transform(df), columns=df.columns)
print(df_filled.round(2))Indicator Columns for Missing Data
Rather than hiding the fact that a value was imputed, it is good practice to add a binary indicator column that flags which rows had missing values before imputation. This lets downstream models learn from the missingness pattern itself — sometimes whether a value is missing is as predictive as the value itself.
import pandas as pd
import numpy as np
df = pd.DataFrame({'salary': [50000, np.nan, 70000, np.nan, 90000]})
# Add indicator before filling
df['salary_was_missing'] = df['salary'].isna().astype(int)
# Then fill
df['salary'] = df['salary'].fillna(df['salary'].median())
print(df)
# salary salary_was_missing
# 0 50000.0 0
# 1 70000.0 1
# 2 70000.0 0
# 3 70000.0 1
# 4 90000.0 0Imputation and Data Leakage
A critical rule in machine learning pipelines: compute imputation statistics (mean, median, mode) on the training set only, then apply the same values to the test set. Computing statistics on the full dataset before splitting causes data leakage — the model indirectly sees test data during training, inflating performance estimates. Always fit imputers on training data and transform both train and test.
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
df = pd.DataFrame({'feature': [1, 2, np.nan, 4, np.nan, 6, 7, 8]})
train, test = train_test_split(df, test_size=0.25, random_state=0)
# Fit ONLY on training data
imp = SimpleImputer(strategy='mean')
train['feature'] = imp.fit_transform(train[['feature']])
# Transform test using training statistics
test['feature'] = imp.transform(test[['feature']])
print('Train mean used:', imp.statistics_[0])Comparing Imputation Strategies Visually
After applying multiple imputation methods, compare their effect by plotting the distribution of the original and imputed columns side by side. A good imputation should preserve the shape (mean, variance, skewness) of the original distribution as closely as possible. Mean imputation narrows the distribution; KNN and MICE tend to preserve it better.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
original = np.random.exponential(scale=5, size=200)
with_nan = original.copy()
with_nan[np.random.choice(200, 40, replace=False)] = np.nan
s = pd.Series(with_nan)
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
s.dropna().hist(ax=axes[0], bins=20, title='Original (no NaN)')
s.fillna(s.mean()).hist(ax=axes[1], bins=20, title='Mean-imputed')
plt.tight_layout()
plt.savefig('imputation_comparison.png')
print('Saved comparison chart')Choosing the Right Imputation Strategy
There is no universally best imputation strategy — the right choice depends on context. Use mean/median for simple univariate cases with low missingness. Use ffill/bfill for time series with stable trends. Use KNN or IterativeImputer when features are correlated and you want to leverage relationships. Use domain constants (e.g., 0 for absent, -1 for unknown) when the missing value has a clear business meaning. Always document your choice.
# Decision guide (no runnable code)
#
# Missing < 5% AND data is MCAR? -> Mean/median fill is fine
# Time series with stable trend? -> ffill() with limit
# Features are correlated? -> KNNImputer or IterativeImputer
# Categorical column? -> Mode fill or 'Unknown' sentinel
# Going into ML model? -> Add indicator column + fill
# Research / publication quality? -> Multiple imputation (MICE)
print('Strategy selected based on context')Quick Check
Test your understanding of interpolation and advanced imputation.
Lesson Recap
In this lesson you learned: interpolate() estimates missing values by assuming a smooth trend between known values, method='time' handles unequal DatetimeIndex gaps, and advanced strategies like KNNImputer and IterativeImputer use other columns to inform fills. Always add indicator columns before imputation and fit statistics on the training set only to avoid data leakage. Next up we inspect and convert DataFrame column data types.
Questions Fréquemment Posées
La leçon « Interpolation et imputation avancée » est-elle gratuite ?
Oui — le texte complet de « Interpolation et imputation avancée » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Pandas & NumPy Academy, passe à CoddyKit PRO. Le cours Pandas & NumPy Academy comprend 4 leçons au total.
Qu'est-ce que j'apprendrai dans « Interpolation et imputation avancée » ?
Utilisez interpolate() pour un remplissage lissé fondé sur le temps et comprenez quand l’imputation par la moyenne ou la médiane est appropriée. Tu pratiques Pandas & NumPy Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.
Dois-je avoir de l'expérience pour commencer Pandas & NumPy Academy ?
Aucune expérience préalable n'est requise. Pandas & NumPy Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 4 sur 4.
Combien de temps prend la leçon « Interpolation et imputation avancée » ?
La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.
Peux-tu écrire et exécuter du code dans cette leçon Pandas & NumPy Academy ?
Oui. Chaque leçon Pandas & NumPy Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.
Toutes les leçons de ce cours
- Détecter les valeurs manquantes
- Supprimer les valeurs manquantes
- Remplacer les valeurs manquantes
- Interpolation et imputation avancée