0Pricing
Machine Learning Academy · Lektion

Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf

Lernende simulieren Drift, indem sie ein Eingabe-Feature schrittweise verschieben, berechnen den Population Stability Index (PSI) und die KL-Divergenz und richten schwellenwertbasierte Warnungen ein.

Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf ist eine kostenlose Machine Learning Academy-Lektion auf CoddyKit. Dies ist Lektion 1 von 4. Du kannst die komplette Lektion unten kostenlos lesen – dann übst du sie direkt im Browser mit einem integrierten Code-Editor und einem KI-Tutor rund um die Uhr. Sie ist Teil des Machine Learning Academy-Lernpfads, und dein Fortschritt wird über Web und CoddyKit-App synchronisiert. Der Machine Learning Academy-Kurs umfasst insgesamt 4 Lektionen.

Teile dieser Lektion wurden noch nicht übersetzt und werden auf Englisch angezeigt.

What Is Data Drift?

Data drift (also called covariate shift) occurs when the statistical distribution of input features changes after deployment compared to the distribution seen during training. A fraud detection model trained on 2022 transaction patterns may encounter very different transaction amounts and merchant categories by 2024. The model's learned decision boundaries no longer match the new data distribution, causing silent performance degradation that only becomes visible through monitoring.

Simulating Drift: Gradual Feature Shift

To study drift, we can simulate it by gradually shifting a feature's mean over time. In production, this might represent seasonal changes in user behaviour, economic shifts affecting purchasing power, or evolving fraud patterns. Plotting the feature distribution for each week reveals when the shift becomes statistically significant and should trigger a retraining alert.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)
# Training distribution: income ~ Normal(50000, 10000)
train_income = np.random.normal(50000, 10000, 5000)

# Production weeks 1-12: mean gradually shifts from 50k to 65k
prod_weeks = []
for week in range(1, 13):
    shifted_mean = 50000 + week * 1250  # +1250 per week
    week_data = np.random.normal(shifted_mean, 10000, 500)
    prod_weeks.append({'week': week, 'income': week_data})

print('Training mean:', train_income.mean().round(0))
for pw in [prod_weeks[0], prod_weeks[5], prod_weeks[-1]]:
    print(f'Week {pw["week"]} mean: {pw["income"].mean().round(0)}')

Population Stability Index (PSI)

The Population Stability Index (PSI) is the most widely used metric for detecting feature drift in the financial industry. It compares two distributions by binning them and measuring the difference in bucket proportions. PSI below 0.1 means no significant shift; 0.1-0.2 indicates moderate shift requiring investigation; above 0.2 signals major drift requiring immediate retraining action.

import numpy as np

def compute_psi(reference, current, buckets=10):
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf

    ref_pcts = np.histogram(reference, bins=breakpoints)[0] / len(reference)
    cur_pcts = np.histogram(current, bins=breakpoints)[0] / len(current)

    # Avoid division by zero
    ref_pcts = np.where(ref_pcts == 0, 1e-6, ref_pcts)
    cur_pcts = np.where(cur_pcts == 0, 1e-6, cur_pcts)

    psi = np.sum((cur_pcts - ref_pcts) * np.log(cur_pcts / ref_pcts))
    return round(psi, 4)

train_income = np.random.normal(50000, 10000, 5000)
week6_income = np.random.normal(57500, 10000, 500)
week12_income = np.random.normal(65000, 10000, 500)

print('PSI week 6:', compute_psi(train_income, week6_income))
print('PSI week 12:', compute_psi(train_income, week12_income))

KL Divergence for Drift Measurement

KL Divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from a reference distribution. It is always non-negative and zero only when the distributions are identical. Unlike PSI, KL divergence is asymmetric: D(P||Q) ≠ D(Q||P). For drift detection, compute KL divergence between training histograms and production histograms for each feature, and alert when it exceeds a calibrated threshold.

import numpy as np
from scipy.stats import entropy
from scipy.special import kl_div

def compute_kl_divergence(reference, current, buckets=20):
    bins = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_hist, _ = np.histogram(reference, bins=bins, density=True)
    cur_hist, _ = np.histogram(current, bins=bins, density=True)

    # Add small epsilon to avoid log(0)
    ref_hist = ref_hist + 1e-10
    cur_hist = cur_hist + 1e-10

    kl = entropy(cur_hist, ref_hist)  # KL(current || reference)
    return round(kl, 4)

train = np.random.normal(0, 1, 5000)
current = np.random.normal(0.5, 1.2, 1000)
print('KL divergence:', compute_kl_divergence(train, current))

Statistical Tests: Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov (KS) test is a non-parametric statistical test that measures the maximum difference between two cumulative distribution functions. Use scipy.stats.ks_2samp to compare reference and production samples. The test returns a statistic (larger = more different) and a p-value (smaller = more statistically significant). A p-value below 0.05 indicates statistically significant distributional difference.

import numpy as np
from scipy.stats import ks_2samp

np.random.seed(42)
train = np.random.normal(50000, 10000, 5000)

week_results = []
for week in range(1, 13):
    prod_sample = np.random.normal(50000 + week * 1250, 10000, 500)
    stat, p_value = ks_2samp(train, prod_sample)
    week_results.append((week, round(stat, 4), round(p_value, 4)))
    alert = 'DRIFT ALERT' if p_value < 0.05 else 'OK'
    print(f'Week {week:2d}: KS={stat:.4f} p={p_value:.4f} {alert}')

Monitoring Multiple Features

Production datasets have many features, and drift can occur in any of them. Monitor all input features by computing drift scores for each column every monitoring period. Use a heatmap to visualise PSI or KS-statistic values across features over time. Features with consistently high drift scores are the primary suspects when model performance degrades and should be investigated first during root-cause analysis.

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

# Simulate reference and production with drift in some features
np.random.seed(42)
n_features = 5
reference = pd.DataFrame(
    np.random.normal(0, 1, (5000, n_features)),
    columns=[f'feature_{i}' for i in range(n_features)]
)
# Introduce drift in features 1 and 3
production = pd.DataFrame(
    np.random.normal([0, 1.5, 0, 2.0, 0], 1, (1000, n_features)),
    columns=reference.columns
)

drift_report = {}
for col in reference.columns:
    stat, p_val = ks_2samp(reference[col], production[col])
    drift_report[col] = {'ks_stat': round(stat, 4), 'p_value': round(p_val, 4),
                          'drift': p_val < 0.05}

for feature, result in drift_report.items():
    status = '*** DRIFT ***' if result['drift'] else 'stable'
    print(f'{feature}: KS={result["ks_stat"]} {status}')

Setting Alert Thresholds

Alert thresholds should be calibrated on historical data, not chosen arbitrarily. A common approach: compute the drift metric between your training set and multiple held-out validation splits to establish a baseline distribution of scores under no-drift conditions. Set the alert threshold at the 99th percentile of this baseline — any production score above this level is statistically unlikely to arise from the same distribution, indicating real drift.

import numpy as np
from scipy.stats import ks_2samp

train = np.random.normal(0, 1, 5000)

# Calibrate: compute KS statistic between train and 100 random validation splits
calibration_scores = []
for _ in range(100):
    val_sample = np.random.normal(0, 1, 500)  # same distribution
    stat, _ = ks_2samp(train, val_sample)
    calibration_scores.append(stat)

threshold_99 = np.percentile(calibration_scores, 99)
print(f'No-drift KS scores -- mean: {np.mean(calibration_scores):.4f}')
print(f'99th percentile threshold: {threshold_99:.4f}')

# Any production score above this triggers an alert
prod_sample_drifted = np.random.normal(0.5, 1, 500)  # shifted
stat, _ = ks_2samp(train, prod_sample_drifted)
print(f'Production KS: {stat:.4f} -- ALERT: {stat > threshold_99}')

Drift in Categorical Features

For categorical features, drift means the frequency distribution of categories changes. A country field might shift from 80% USA in training to 60% USA in production as the product expands globally. Use Chi-squared tests or Jensen-Shannon divergence to compare categorical distributions. New categories appearing in production that were absent from training are a special case — they result in KeyError in label encoders that were not fitted with handle_unknown='ignore'.

import numpy as np
from scipy.stats import chi2_contingency

# Training distribution
train_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.7, 0.15, 0.1, 0.05], size=5000
)
# Production: more EU traffic
prod_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.5, 0.2, 0.2, 0.1], size=1000
)

categories = ['US', 'GB', 'DE', 'FR']
train_counts = [np.sum(train_countries == c) for c in categories]
prod_counts = [np.sum(prod_countries == c) for c in categories]

contingency = np.array([train_counts, prod_counts])
chi2, p_value, dof, _ = chi2_contingency(contingency)
print(f'Chi-squared: {chi2:.2f}, p-value: {p_value:.4f}')
print('Categorical drift detected:', p_value < 0.05)

Logging Drift Metrics Over Time

Drift monitoring is only useful if you log metrics consistently over time and visualise trends. Store weekly drift scores in a time-series database (InfluxDB, PostgreSQL, or even a CSV file) and plot them with a rolling window. A sudden spike in drift score pinpoints the exact week that a data pipeline changed or an external event shifted user behaviour, helping the team perform faster root-cause analysis.

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

records = []
base_date = datetime(2024, 1, 1)

for week in range(12):
    prod_sample = np.random.normal(week * 0.1, 1, 500)  # gradual drift
    from scipy.stats import ks_2samp
    train_sample = np.random.normal(0, 1, 5000)
    stat, p = ks_2samp(train_sample, prod_sample)
    records.append({
        'date': base_date + timedelta(weeks=week),
        'ks_stat': round(stat, 4),
        'p_value': round(p, 4),
        'alert': p < 0.05
    })

df = pd.DataFrame(records)
print(df.to_string(index=False))
# In production: df.to_sql('drift_log', engine, if_exists='append')

What to Do When Drift Is Detected

When drift is detected, there are several response strategies depending on severity. Minor drift: increase monitoring frequency and investigate the cause. Moderate drift: trigger retraining with the most recent data window. Severe drift: consider whether the old feature engineering is still valid for the new distribution, potentially requiring feature redesign. Always document the drift event and the business context (e.g., a marketing campaign, a platform change) that caused it.

def respond_to_drift(psi_score, ks_p_value):
    if psi_score < 0.1 and ks_p_value > 0.05:
        return 'No action needed. All features stable.'

    elif psi_score < 0.2 and ks_p_value > 0.01:
        return ('Moderate drift detected. '
                'Increase monitoring to daily. '
                'Schedule retraining for next cycle.')

    else:
        return ('SEVERE DRIFT. '
                'Trigger emergency retraining now. '
                'Consider feature engineering review. '
                'Alert data engineering team.')

# Example
print(respond_to_drift(0.05, 0.3))   # stable
print(respond_to_drift(0.15, 0.04))  # moderate
print(respond_to_drift(0.35, 0.001)) # severe

Visualising Feature Drift Over Time

Visualising drift as a time series helps stakeholders understand severity and timing. Plot the PSI or KS statistic for each monitored feature on a rolling weekly basis. When a feature line crosses the alert threshold, annotate the chart with the date and the nature of the business event that caused the shift (e.g., a marketing campaign, a pricing change, or a seasonal pattern). This converts raw statistics into actionable business intelligence.

import matplotlib.pyplot as plt
import numpy as np

weeks = list(range(1, 13))
psi_feature_A = [0.01, 0.02, 0.03, 0.05, 0.07, 0.11, 0.18, 0.22, 0.25, 0.24, 0.23, 0.26]
psi_feature_B = [0.01, 0.01, 0.02, 0.01, 0.02, 0.03, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08]

plt.figure(figsize=(10, 4))
plt.plot(weeks, psi_feature_A, marker='o', label='amount_usd (drifting)')
plt.plot(weeks, psi_feature_B, marker='s', label='merchant_category (stable)')
plt.axhline(0.1, color='orange', linestyle='--', label='Mild drift threshold')
plt.axhline(0.25, color='red', linestyle='--', label='Severe drift threshold')
plt.annotate('Pricing change', xy=(6, 0.11), xytext=(6, 0.16), arrowprops=dict(arrowstyle='->'))
plt.xlabel('Week')
plt.ylabel('PSI')
plt.legend()
plt.title('PSI Trend — Feature-Level Drift Monitoring')
plt.tight_layout()
plt.savefig('psi_trend.png', dpi=150)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: data drift is the shift in input feature distributions between training time and production time, causing silent model degradation, PSI and KS tests provide quantitative drift scores with industry-standard alert thresholds (PSI > 0.2 = major drift), and drift monitoring should cover all input features and log scores over time to enable trend analysis and root-cause identification. Next up we distinguish concept drift — when the relationship between inputs and the target label itself changes, which is a more subtle and dangerous phenomenon.

Häufig gestellte Fragen

Ist die Lektion „Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf“ kostenlos?

Ja — der vollständige Text von „Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf“ ist hier im Web kostenlos zu lesen. Um sie interaktiv zu üben (integrierter Code-Editor und 24/7 KI-Tutor) und den Rest des Machine Learning Academy-Kurses freizuschalten, upgrade auf CoddyKit PRO. Der Machine Learning Academy-Kurs umfasst insgesamt 4 Lektionen.

Was lerne ich in „Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf“?

Lernende simulieren Drift, indem sie ein Eingabe-Feature schrittweise verschieben, berechnen den Population Stability Index (PSI) und die KL-Divergenz und richten schwellenwertbasierte Warnungen ein. Du übst Machine Learning Academy mit praktischem Code, den du direkt im Browser ausführst, und ein 24/7 KI-Tutor beantwortet deine Fragen während du die Lektion bearbeitest.

Brauche ich Erfahrung, um Machine Learning Academy zu starten?

Keine Vorkenntnisse erforderlich. Machine Learning Academy auf CoddyKit ist für Anfänger bis fortgeschrittene Lernende strukturiert, sodass du hier starten oder von Anfang an beginnen und in deinem eigenen Tempo voranschreiten kannst. Dies ist Lektion 1 von 4.

Wie lange dauert die Lektion „Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf“?

Die meisten CoddyKit-Lektionen dauern etwa 5–10 Minuten. Jede ist kompakt und interaktiv, sodass du stetig Fortschritte machst und genau dort weitermachst, wo du aufgehört hast – im Web und in der App.

Kann ich in dieser Machine Learning Academy-Lektion Code schreiben und ausführen?

Ja. Jede Machine Learning Academy-Lektion enthält einen integrierten Code-Editor, sodass du echten Code direkt in deinem Browser schreibst und ausführst und sofort KI-Feedback erhältst — ohne lokale Einrichtung erforderlich.

Alle Lektionen in diesem Kurs

  1. Data Drift: Verschiebungen der Feature-Verteilung im Zeitverlauf
  2. Concept Drift: Wenn sich die Beziehung zwischen X und Y ändert
  3. Vorhersageverteilungen und Konfidenzwerte überwachen
  4. Eine Drift-Warnpipeline mit Evidently AI erstellen
← Zurück zu Machine Learning Academy