0Pricing
Machine Learning Academy · Lesson

Data Drift: Feature Distribution Shifts Over Time

Learners will simulate drift by gradually shifting an input feature, compute the Population Stability Index (PSI) and KL divergence, and set threshold-based alerts.

Data Drift: Feature Distribution Shifts Over Time is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Data Drift?

Data drift (also called covariate shift) occurs when the statistical distribution of input features changes after deployment compared to the distribution seen during training. A fraud detection model trained on 2022 transaction patterns may encounter very different transaction amounts and merchant categories by 2024. The model's learned decision boundaries no longer match the new data distribution, causing silent performance degradation that only becomes visible through monitoring.

Simulating Drift: Gradual Feature Shift

To study drift, we can simulate it by gradually shifting a feature's mean over time. In production, this might represent seasonal changes in user behaviour, economic shifts affecting purchasing power, or evolving fraud patterns. Plotting the feature distribution for each week reveals when the shift becomes statistically significant and should trigger a retraining alert.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)
# Training distribution: income ~ Normal(50000, 10000)
train_income = np.random.normal(50000, 10000, 5000)

# Production weeks 1-12: mean gradually shifts from 50k to 65k
prod_weeks = []
for week in range(1, 13):
    shifted_mean = 50000 + week * 1250  # +1250 per week
    week_data = np.random.normal(shifted_mean, 10000, 500)
    prod_weeks.append({'week': week, 'income': week_data})

print('Training mean:', train_income.mean().round(0))
for pw in [prod_weeks[0], prod_weeks[5], prod_weeks[-1]]:
    print(f'Week {pw["week"]} mean: {pw["income"].mean().round(0)}')

Population Stability Index (PSI)

The Population Stability Index (PSI) is the most widely used metric for detecting feature drift in the financial industry. It compares two distributions by binning them and measuring the difference in bucket proportions. PSI below 0.1 means no significant shift; 0.1-0.2 indicates moderate shift requiring investigation; above 0.2 signals major drift requiring immediate retraining action.

import numpy as np

def compute_psi(reference, current, buckets=10):
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf

    ref_pcts = np.histogram(reference, bins=breakpoints)[0] / len(reference)
    cur_pcts = np.histogram(current, bins=breakpoints)[0] / len(current)

    # Avoid division by zero
    ref_pcts = np.where(ref_pcts == 0, 1e-6, ref_pcts)
    cur_pcts = np.where(cur_pcts == 0, 1e-6, cur_pcts)

    psi = np.sum((cur_pcts - ref_pcts) * np.log(cur_pcts / ref_pcts))
    return round(psi, 4)

train_income = np.random.normal(50000, 10000, 5000)
week6_income = np.random.normal(57500, 10000, 500)
week12_income = np.random.normal(65000, 10000, 500)

print('PSI week 6:', compute_psi(train_income, week6_income))
print('PSI week 12:', compute_psi(train_income, week12_income))

KL Divergence for Drift Measurement

KL Divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from a reference distribution. It is always non-negative and zero only when the distributions are identical. Unlike PSI, KL divergence is asymmetric: D(P||Q) ≠ D(Q||P). For drift detection, compute KL divergence between training histograms and production histograms for each feature, and alert when it exceeds a calibrated threshold.

import numpy as np
from scipy.stats import entropy
from scipy.special import kl_div

def compute_kl_divergence(reference, current, buckets=20):
    bins = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_hist, _ = np.histogram(reference, bins=bins, density=True)
    cur_hist, _ = np.histogram(current, bins=bins, density=True)

    # Add small epsilon to avoid log(0)
    ref_hist = ref_hist + 1e-10
    cur_hist = cur_hist + 1e-10

    kl = entropy(cur_hist, ref_hist)  # KL(current || reference)
    return round(kl, 4)

train = np.random.normal(0, 1, 5000)
current = np.random.normal(0.5, 1.2, 1000)
print('KL divergence:', compute_kl_divergence(train, current))

Statistical Tests: Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov (KS) test is a non-parametric statistical test that measures the maximum difference between two cumulative distribution functions. Use scipy.stats.ks_2samp to compare reference and production samples. The test returns a statistic (larger = more different) and a p-value (smaller = more statistically significant). A p-value below 0.05 indicates statistically significant distributional difference.

import numpy as np
from scipy.stats import ks_2samp

np.random.seed(42)
train = np.random.normal(50000, 10000, 5000)

week_results = []
for week in range(1, 13):
    prod_sample = np.random.normal(50000 + week * 1250, 10000, 500)
    stat, p_value = ks_2samp(train, prod_sample)
    week_results.append((week, round(stat, 4), round(p_value, 4)))
    alert = 'DRIFT ALERT' if p_value < 0.05 else 'OK'
    print(f'Week {week:2d}: KS={stat:.4f} p={p_value:.4f} {alert}')

Monitoring Multiple Features

Production datasets have many features, and drift can occur in any of them. Monitor all input features by computing drift scores for each column every monitoring period. Use a heatmap to visualise PSI or KS-statistic values across features over time. Features with consistently high drift scores are the primary suspects when model performance degrades and should be investigated first during root-cause analysis.

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

# Simulate reference and production with drift in some features
np.random.seed(42)
n_features = 5
reference = pd.DataFrame(
    np.random.normal(0, 1, (5000, n_features)),
    columns=[f'feature_{i}' for i in range(n_features)]
)
# Introduce drift in features 1 and 3
production = pd.DataFrame(
    np.random.normal([0, 1.5, 0, 2.0, 0], 1, (1000, n_features)),
    columns=reference.columns
)

drift_report = {}
for col in reference.columns:
    stat, p_val = ks_2samp(reference[col], production[col])
    drift_report[col] = {'ks_stat': round(stat, 4), 'p_value': round(p_val, 4),
                          'drift': p_val < 0.05}

for feature, result in drift_report.items():
    status = '*** DRIFT ***' if result['drift'] else 'stable'
    print(f'{feature}: KS={result["ks_stat"]} {status}')

Setting Alert Thresholds

Alert thresholds should be calibrated on historical data, not chosen arbitrarily. A common approach: compute the drift metric between your training set and multiple held-out validation splits to establish a baseline distribution of scores under no-drift conditions. Set the alert threshold at the 99th percentile of this baseline — any production score above this level is statistically unlikely to arise from the same distribution, indicating real drift.

import numpy as np
from scipy.stats import ks_2samp

train = np.random.normal(0, 1, 5000)

# Calibrate: compute KS statistic between train and 100 random validation splits
calibration_scores = []
for _ in range(100):
    val_sample = np.random.normal(0, 1, 500)  # same distribution
    stat, _ = ks_2samp(train, val_sample)
    calibration_scores.append(stat)

threshold_99 = np.percentile(calibration_scores, 99)
print(f'No-drift KS scores -- mean: {np.mean(calibration_scores):.4f}')
print(f'99th percentile threshold: {threshold_99:.4f}')

# Any production score above this triggers an alert
prod_sample_drifted = np.random.normal(0.5, 1, 500)  # shifted
stat, _ = ks_2samp(train, prod_sample_drifted)
print(f'Production KS: {stat:.4f} -- ALERT: {stat > threshold_99}')

Drift in Categorical Features

For categorical features, drift means the frequency distribution of categories changes. A country field might shift from 80% USA in training to 60% USA in production as the product expands globally. Use Chi-squared tests or Jensen-Shannon divergence to compare categorical distributions. New categories appearing in production that were absent from training are a special case — they result in KeyError in label encoders that were not fitted with handle_unknown='ignore'.

import numpy as np
from scipy.stats import chi2_contingency

# Training distribution
train_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.7, 0.15, 0.1, 0.05], size=5000
)
# Production: more EU traffic
prod_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.5, 0.2, 0.2, 0.1], size=1000
)

categories = ['US', 'GB', 'DE', 'FR']
train_counts = [np.sum(train_countries == c) for c in categories]
prod_counts = [np.sum(prod_countries == c) for c in categories]

contingency = np.array([train_counts, prod_counts])
chi2, p_value, dof, _ = chi2_contingency(contingency)
print(f'Chi-squared: {chi2:.2f}, p-value: {p_value:.4f}')
print('Categorical drift detected:', p_value < 0.05)

Logging Drift Metrics Over Time

Drift monitoring is only useful if you log metrics consistently over time and visualise trends. Store weekly drift scores in a time-series database (InfluxDB, PostgreSQL, or even a CSV file) and plot them with a rolling window. A sudden spike in drift score pinpoints the exact week that a data pipeline changed or an external event shifted user behaviour, helping the team perform faster root-cause analysis.

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

records = []
base_date = datetime(2024, 1, 1)

for week in range(12):
    prod_sample = np.random.normal(week * 0.1, 1, 500)  # gradual drift
    from scipy.stats import ks_2samp
    train_sample = np.random.normal(0, 1, 5000)
    stat, p = ks_2samp(train_sample, prod_sample)
    records.append({
        'date': base_date + timedelta(weeks=week),
        'ks_stat': round(stat, 4),
        'p_value': round(p, 4),
        'alert': p < 0.05
    })

df = pd.DataFrame(records)
print(df.to_string(index=False))
# In production: df.to_sql('drift_log', engine, if_exists='append')

What to Do When Drift Is Detected

When drift is detected, there are several response strategies depending on severity. Minor drift: increase monitoring frequency and investigate the cause. Moderate drift: trigger retraining with the most recent data window. Severe drift: consider whether the old feature engineering is still valid for the new distribution, potentially requiring feature redesign. Always document the drift event and the business context (e.g., a marketing campaign, a platform change) that caused it.

def respond_to_drift(psi_score, ks_p_value):
    if psi_score < 0.1 and ks_p_value > 0.05:
        return 'No action needed. All features stable.'

    elif psi_score < 0.2 and ks_p_value > 0.01:
        return ('Moderate drift detected. '
                'Increase monitoring to daily. '
                'Schedule retraining for next cycle.')

    else:
        return ('SEVERE DRIFT. '
                'Trigger emergency retraining now. '
                'Consider feature engineering review. '
                'Alert data engineering team.')

# Example
print(respond_to_drift(0.05, 0.3))   # stable
print(respond_to_drift(0.15, 0.04))  # moderate
print(respond_to_drift(0.35, 0.001)) # severe

Visualising Feature Drift Over Time

Visualising drift as a time series helps stakeholders understand severity and timing. Plot the PSI or KS statistic for each monitored feature on a rolling weekly basis. When a feature line crosses the alert threshold, annotate the chart with the date and the nature of the business event that caused the shift (e.g., a marketing campaign, a pricing change, or a seasonal pattern). This converts raw statistics into actionable business intelligence.

import matplotlib.pyplot as plt
import numpy as np

weeks = list(range(1, 13))
psi_feature_A = [0.01, 0.02, 0.03, 0.05, 0.07, 0.11, 0.18, 0.22, 0.25, 0.24, 0.23, 0.26]
psi_feature_B = [0.01, 0.01, 0.02, 0.01, 0.02, 0.03, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08]

plt.figure(figsize=(10, 4))
plt.plot(weeks, psi_feature_A, marker='o', label='amount_usd (drifting)')
plt.plot(weeks, psi_feature_B, marker='s', label='merchant_category (stable)')
plt.axhline(0.1, color='orange', linestyle='--', label='Mild drift threshold')
plt.axhline(0.25, color='red', linestyle='--', label='Severe drift threshold')
plt.annotate('Pricing change', xy=(6, 0.11), xytext=(6, 0.16), arrowprops=dict(arrowstyle='->'))
plt.xlabel('Week')
plt.ylabel('PSI')
plt.legend()
plt.title('PSI Trend — Feature-Level Drift Monitoring')
plt.tight_layout()
plt.savefig('psi_trend.png', dpi=150)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: data drift is the shift in input feature distributions between training time and production time, causing silent model degradation, PSI and KS tests provide quantitative drift scores with industry-standard alert thresholds (PSI > 0.2 = major drift), and drift monitoring should cover all input features and log scores over time to enable trend analysis and root-cause identification. Next up we distinguish concept drift — when the relationship between inputs and the target label itself changes, which is a more subtle and dangerous phenomenon.

Frequently asked questions

Is the “Data Drift: Feature Distribution Shifts Over Time” lesson free?

Yes — the full text of “Data Drift: Feature Distribution Shifts Over Time” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Data Drift: Feature Distribution Shifts Over Time”?

Learners will simulate drift by gradually shifting an input feature, compute the Population Stability Index (PSI) and KL divergence, and set threshold-based alerts. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Data Drift: Feature Distribution Shifts Over Time” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Data Drift: Feature Distribution Shifts Over Time
  2. Concept Drift: When the Relationship Between X and Y Changes
  3. Monitoring Prediction Distributions and Confidence Scores
  4. Building a Drift Alert Pipeline with Evidently AI
← Back to Machine Learning Academy