Machine Learning Academy · レッスン

データドリフト：時間経過に伴う特徴量分布の変化

入力特徴量を徐々に変化させてドリフトをシミュレートし、Population Stability Index（PSI）とKLダイバージェンスを計算して、しきい値ベースのアラートを設定します。

「データドリフト：時間経過に伴う特徴量分布の変化」はCoddyKit上の無料Machine Learning Academyレッスンです。これはレッスン1/4です。下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。これはMachine Learning Academy学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Machine Learning Academyコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

What Is Data Drift?

Data drift (also called covariate shift) occurs when the statistical distribution of input features changes after deployment compared to the distribution seen during training. A fraud detection model trained on 2022 transaction patterns may encounter very different transaction amounts and merchant categories by 2024. The model's learned decision boundaries no longer match the new data distribution, causing silent performance degradation that only becomes visible through monitoring.

Simulating Drift: Gradual Feature Shift

To study drift, we can simulate it by gradually shifting a feature's mean over time. In production, this might represent seasonal changes in user behaviour, economic shifts affecting purchasing power, or evolving fraud patterns. Plotting the feature distribution for each week reveals when the shift becomes statistically significant and should trigger a retraining alert.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)
# Training distribution: income ~ Normal(50000, 10000)
train_income = np.random.normal(50000, 10000, 5000)

# Production weeks 1-12: mean gradually shifts from 50k to 65k
prod_weeks = []
for week in range(1, 13):
    shifted_mean = 50000 + week * 1250  # +1250 per week
    week_data = np.random.normal(shifted_mean, 10000, 500)
    prod_weeks.append({'week': week, 'income': week_data})

print('Training mean:', train_income.mean().round(0))
for pw in [prod_weeks[0], prod_weeks[5], prod_weeks[-1]]:
    print(f'Week {pw["week"]} mean: {pw["income"].mean().round(0)}')

Population Stability Index (PSI)

The Population Stability Index (PSI) is the most widely used metric for detecting feature drift in the financial industry. It compares two distributions by binning them and measuring the difference in bucket proportions. PSI below 0.1 means no significant shift; 0.1-0.2 indicates moderate shift requiring investigation; above 0.2 signals major drift requiring immediate retraining action.

import numpy as np

def compute_psi(reference, current, buckets=10):
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf

    ref_pcts = np.histogram(reference, bins=breakpoints)[0] / len(reference)
    cur_pcts = np.histogram(current, bins=breakpoints)[0] / len(current)

    # Avoid division by zero
    ref_pcts = np.where(ref_pcts == 0, 1e-6, ref_pcts)
    cur_pcts = np.where(cur_pcts == 0, 1e-6, cur_pcts)

    psi = np.sum((cur_pcts - ref_pcts) * np.log(cur_pcts / ref_pcts))
    return round(psi, 4)

train_income = np.random.normal(50000, 10000, 5000)
week6_income = np.random.normal(57500, 10000, 500)
week12_income = np.random.normal(65000, 10000, 500)

print('PSI week 6:', compute_psi(train_income, week6_income))
print('PSI week 12:', compute_psi(train_income, week12_income))

KL Divergence for Drift Measurement

KL Divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from a reference distribution. It is always non-negative and zero only when the distributions are identical. Unlike PSI, KL divergence is asymmetric: D(P||Q) ≠ D(Q||P). For drift detection, compute KL divergence between training histograms and production histograms for each feature, and alert when it exceeds a calibrated threshold.

import numpy as np
from scipy.stats import entropy
from scipy.special import kl_div

def compute_kl_divergence(reference, current, buckets=20):
    bins = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_hist, _ = np.histogram(reference, bins=bins, density=True)
    cur_hist, _ = np.histogram(current, bins=bins, density=True)

    # Add small epsilon to avoid log(0)
    ref_hist = ref_hist + 1e-10
    cur_hist = cur_hist + 1e-10

    kl = entropy(cur_hist, ref_hist)  # KL(current || reference)
    return round(kl, 4)

train = np.random.normal(0, 1, 5000)
current = np.random.normal(0.5, 1.2, 1000)
print('KL divergence:', compute_kl_divergence(train, current))

Statistical Tests: Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov (KS) test is a non-parametric statistical test that measures the maximum difference between two cumulative distribution functions. Use scipy.stats.ks_2samp to compare reference and production samples. The test returns a statistic (larger = more different) and a p-value (smaller = more statistically significant). A p-value below 0.05 indicates statistically significant distributional difference.

import numpy as np
from scipy.stats import ks_2samp

np.random.seed(42)
train = np.random.normal(50000, 10000, 5000)

week_results = []
for week in range(1, 13):
    prod_sample = np.random.normal(50000 + week * 1250, 10000, 500)
    stat, p_value = ks_2samp(train, prod_sample)
    week_results.append((week, round(stat, 4), round(p_value, 4)))
    alert = 'DRIFT ALERT' if p_value < 0.05 else 'OK'
    print(f'Week {week:2d}: KS={stat:.4f} p={p_value:.4f} {alert}')

Monitoring Multiple Features

Production datasets have many features, and drift can occur in any of them. Monitor all input features by computing drift scores for each column every monitoring period. Use a heatmap to visualise PSI or KS-statistic values across features over time. Features with consistently high drift scores are the primary suspects when model performance degrades and should be investigated first during root-cause analysis.

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

# Simulate reference and production with drift in some features
np.random.seed(42)
n_features = 5
reference = pd.DataFrame(
    np.random.normal(0, 1, (5000, n_features)),
    columns=[f'feature_{i}' for i in range(n_features)]
)
# Introduce drift in features 1 and 3
production = pd.DataFrame(
    np.random.normal([0, 1.5, 0, 2.0, 0], 1, (1000, n_features)),
    columns=reference.columns
)

drift_report = {}
for col in reference.columns:
    stat, p_val = ks_2samp(reference[col], production[col])
    drift_report[col] = {'ks_stat': round(stat, 4), 'p_value': round(p_val, 4),
                          'drift': p_val < 0.05}

for feature, result in drift_report.items():
    status = '*** DRIFT ***' if result['drift'] else 'stable'
    print(f'{feature}: KS={result["ks_stat"]} {status}')

Setting Alert Thresholds

Alert thresholds should be calibrated on historical data, not chosen arbitrarily. A common approach: compute the drift metric between your training set and multiple held-out validation splits to establish a baseline distribution of scores under no-drift conditions. Set the alert threshold at the 99th percentile of this baseline — any production score above this level is statistically unlikely to arise from the same distribution, indicating real drift.

import numpy as np
from scipy.stats import ks_2samp

train = np.random.normal(0, 1, 5000)

# Calibrate: compute KS statistic between train and 100 random validation splits
calibration_scores = []
for _ in range(100):
    val_sample = np.random.normal(0, 1, 500)  # same distribution
    stat, _ = ks_2samp(train, val_sample)
    calibration_scores.append(stat)

threshold_99 = np.percentile(calibration_scores, 99)
print(f'No-drift KS scores -- mean: {np.mean(calibration_scores):.4f}')
print(f'99th percentile threshold: {threshold_99:.4f}')

# Any production score above this triggers an alert
prod_sample_drifted = np.random.normal(0.5, 1, 500)  # shifted
stat, _ = ks_2samp(train, prod_sample_drifted)
print(f'Production KS: {stat:.4f} -- ALERT: {stat > threshold_99}')

Drift in Categorical Features

For categorical features, drift means the frequency distribution of categories changes. A country field might shift from 80% USA in training to 60% USA in production as the product expands globally. Use Chi-squared tests or Jensen-Shannon divergence to compare categorical distributions. New categories appearing in production that were absent from training are a special case — they result in KeyError in label encoders that were not fitted with handle_unknown='ignore'.

import numpy as np
from scipy.stats import chi2_contingency

# Training distribution
train_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.7, 0.15, 0.1, 0.05], size=5000
)
# Production: more EU traffic
prod_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.5, 0.2, 0.2, 0.1], size=1000
)

categories = ['US', 'GB', 'DE', 'FR']
train_counts = [np.sum(train_countries == c) for c in categories]
prod_counts = [np.sum(prod_countries == c) for c in categories]

contingency = np.array([train_counts, prod_counts])
chi2, p_value, dof, _ = chi2_contingency(contingency)
print(f'Chi-squared: {chi2:.2f}, p-value: {p_value:.4f}')
print('Categorical drift detected:', p_value < 0.05)

Logging Drift Metrics Over Time

Drift monitoring is only useful if you log metrics consistently over time and visualise trends. Store weekly drift scores in a time-series database (InfluxDB, PostgreSQL, or even a CSV file) and plot them with a rolling window. A sudden spike in drift score pinpoints the exact week that a data pipeline changed or an external event shifted user behaviour, helping the team perform faster root-cause analysis.

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

records = []
base_date = datetime(2024, 1, 1)

for week in range(12):
    prod_sample = np.random.normal(week * 0.1, 1, 500)  # gradual drift
    from scipy.stats import ks_2samp
    train_sample = np.random.normal(0, 1, 5000)
    stat, p = ks_2samp(train_sample, prod_sample)
    records.append({
        'date': base_date + timedelta(weeks=week),
        'ks_stat': round(stat, 4),
        'p_value': round(p, 4),
        'alert': p < 0.05
    })

df = pd.DataFrame(records)
print(df.to_string(index=False))
# In production: df.to_sql('drift_log', engine, if_exists='append')

What to Do When Drift Is Detected

When drift is detected, there are several response strategies depending on severity. Minor drift: increase monitoring frequency and investigate the cause. Moderate drift: trigger retraining with the most recent data window. Severe drift: consider whether the old feature engineering is still valid for the new distribution, potentially requiring feature redesign. Always document the drift event and the business context (e.g., a marketing campaign, a platform change) that caused it.

def respond_to_drift(psi_score, ks_p_value):
    if psi_score < 0.1 and ks_p_value > 0.05:
        return 'No action needed. All features stable.'

    elif psi_score < 0.2 and ks_p_value > 0.01:
        return ('Moderate drift detected. '
                'Increase monitoring to daily. '
                'Schedule retraining for next cycle.')

    else:
        return ('SEVERE DRIFT. '
                'Trigger emergency retraining now. '
                'Consider feature engineering review. '
                'Alert data engineering team.')

# Example
print(respond_to_drift(0.05, 0.3))   # stable
print(respond_to_drift(0.15, 0.04))  # moderate
print(respond_to_drift(0.35, 0.001)) # severe

Visualising Feature Drift Over Time

Visualising drift as a time series helps stakeholders understand severity and timing. Plot the PSI or KS statistic for each monitored feature on a rolling weekly basis. When a feature line crosses the alert threshold, annotate the chart with the date and the nature of the business event that caused the shift (e.g., a marketing campaign, a pricing change, or a seasonal pattern). This converts raw statistics into actionable business intelligence.

import matplotlib.pyplot as plt
import numpy as np

weeks = list(range(1, 13))
psi_feature_A = [0.01, 0.02, 0.03, 0.05, 0.07, 0.11, 0.18, 0.22, 0.25, 0.24, 0.23, 0.26]
psi_feature_B = [0.01, 0.01, 0.02, 0.01, 0.02, 0.03, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08]

plt.figure(figsize=(10, 4))
plt.plot(weeks, psi_feature_A, marker='o', label='amount_usd (drifting)')
plt.plot(weeks, psi_feature_B, marker='s', label='merchant_category (stable)')
plt.axhline(0.1, color='orange', linestyle='--', label='Mild drift threshold')
plt.axhline(0.25, color='red', linestyle='--', label='Severe drift threshold')
plt.annotate('Pricing change', xy=(6, 0.11), xytext=(6, 0.16), arrowprops=dict(arrowstyle='->'))
plt.xlabel('Week')
plt.ylabel('PSI')
plt.legend()
plt.title('PSI Trend — Feature-Level Drift Monitoring')
plt.tight_layout()
plt.savefig('psi_trend.png', dpi=150)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: data drift is the shift in input feature distributions between training time and production time, causing silent model degradation, PSI and KS tests provide quantitative drift scores with industry-standard alert thresholds (PSI > 0.2 = major drift), and drift monitoring should cover all input features and log scores over time to enable trend analysis and root-cause identification. Next up we distinguish concept drift — when the relationship between inputs and the target label itself changes, which is a more subtle and dangerous phenomenon.

よくある質問

「データドリフト：時間経過に伴う特徴量分布の変化」レッスンは無料ですか？

はい。「データドリフト：時間経過に伴う特徴量分布の変化」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Machine Learning Academyコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Machine Learning Academyコースには全4レッスンが含まれています。

「データドリフト：時間経過に伴う特徴量分布の変化」で何を学びますか？

入力特徴量を徐々に変化させてドリフトをシミュレートし、Population Stability Index（PSI）とKLダイバージェンスを計算して、しきい値ベースのアラートを設定します。ブラウザで直接実行するハンズオンコードでMachine Learning Academyを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

Machine Learning Academyを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのMachine Learning Academyは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。これはレッスン1/4です。

「データドリフト：時間経過に伴う特徴量分布の変化」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このMachine Learning Academyレッスンでコードを書いて実行できますか？

はい。すべてのMachine Learning Academyレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

← Machine Learning Academyに戻る