0Pricing
Machine Learning Academy · Pelajaran

Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu

Peserta akan mensimulasikan pergeseran dengan mengubah fitur input secara bertahap, menghitung Population Stability Index (PSI) dan divergensi KL, serta menetapkan peringatan berbasis ambang.

Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 1 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.

What Is Data Drift?

Data drift (also called covariate shift) occurs when the statistical distribution of input features changes after deployment compared to the distribution seen during training. A fraud detection model trained on 2022 transaction patterns may encounter very different transaction amounts and merchant categories by 2024. The model's learned decision boundaries no longer match the new data distribution, causing silent performance degradation that only becomes visible through monitoring.

Simulating Drift: Gradual Feature Shift

To study drift, we can simulate it by gradually shifting a feature's mean over time. In production, this might represent seasonal changes in user behaviour, economic shifts affecting purchasing power, or evolving fraud patterns. Plotting the feature distribution for each week reveals when the shift becomes statistically significant and should trigger a retraining alert.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)
# Training distribution: income ~ Normal(50000, 10000)
train_income = np.random.normal(50000, 10000, 5000)

# Production weeks 1-12: mean gradually shifts from 50k to 65k
prod_weeks = []
for week in range(1, 13):
    shifted_mean = 50000 + week * 1250  # +1250 per week
    week_data = np.random.normal(shifted_mean, 10000, 500)
    prod_weeks.append({'week': week, 'income': week_data})

print('Training mean:', train_income.mean().round(0))
for pw in [prod_weeks[0], prod_weeks[5], prod_weeks[-1]]:
    print(f'Week {pw["week"]} mean: {pw["income"].mean().round(0)}')

Population Stability Index (PSI)

The Population Stability Index (PSI) is the most widely used metric for detecting feature drift in the financial industry. It compares two distributions by binning them and measuring the difference in bucket proportions. PSI below 0.1 means no significant shift; 0.1-0.2 indicates moderate shift requiring investigation; above 0.2 signals major drift requiring immediate retraining action.

import numpy as np

def compute_psi(reference, current, buckets=10):
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf

    ref_pcts = np.histogram(reference, bins=breakpoints)[0] / len(reference)
    cur_pcts = np.histogram(current, bins=breakpoints)[0] / len(current)

    # Avoid division by zero
    ref_pcts = np.where(ref_pcts == 0, 1e-6, ref_pcts)
    cur_pcts = np.where(cur_pcts == 0, 1e-6, cur_pcts)

    psi = np.sum((cur_pcts - ref_pcts) * np.log(cur_pcts / ref_pcts))
    return round(psi, 4)

train_income = np.random.normal(50000, 10000, 5000)
week6_income = np.random.normal(57500, 10000, 500)
week12_income = np.random.normal(65000, 10000, 500)

print('PSI week 6:', compute_psi(train_income, week6_income))
print('PSI week 12:', compute_psi(train_income, week12_income))

KL Divergence for Drift Measurement

KL Divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from a reference distribution. It is always non-negative and zero only when the distributions are identical. Unlike PSI, KL divergence is asymmetric: D(P||Q) ≠ D(Q||P). For drift detection, compute KL divergence between training histograms and production histograms for each feature, and alert when it exceeds a calibrated threshold.

import numpy as np
from scipy.stats import entropy
from scipy.special import kl_div

def compute_kl_divergence(reference, current, buckets=20):
    bins = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_hist, _ = np.histogram(reference, bins=bins, density=True)
    cur_hist, _ = np.histogram(current, bins=bins, density=True)

    # Add small epsilon to avoid log(0)
    ref_hist = ref_hist + 1e-10
    cur_hist = cur_hist + 1e-10

    kl = entropy(cur_hist, ref_hist)  # KL(current || reference)
    return round(kl, 4)

train = np.random.normal(0, 1, 5000)
current = np.random.normal(0.5, 1.2, 1000)
print('KL divergence:', compute_kl_divergence(train, current))

Statistical Tests: Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov (KS) test is a non-parametric statistical test that measures the maximum difference between two cumulative distribution functions. Use scipy.stats.ks_2samp to compare reference and production samples. The test returns a statistic (larger = more different) and a p-value (smaller = more statistically significant). A p-value below 0.05 indicates statistically significant distributional difference.

import numpy as np
from scipy.stats import ks_2samp

np.random.seed(42)
train = np.random.normal(50000, 10000, 5000)

week_results = []
for week in range(1, 13):
    prod_sample = np.random.normal(50000 + week * 1250, 10000, 500)
    stat, p_value = ks_2samp(train, prod_sample)
    week_results.append((week, round(stat, 4), round(p_value, 4)))
    alert = 'DRIFT ALERT' if p_value < 0.05 else 'OK'
    print(f'Week {week:2d}: KS={stat:.4f} p={p_value:.4f} {alert}')

Monitoring Multiple Features

Production datasets have many features, and drift can occur in any of them. Monitor all input features by computing drift scores for each column every monitoring period. Use a heatmap to visualise PSI or KS-statistic values across features over time. Features with consistently high drift scores are the primary suspects when model performance degrades and should be investigated first during root-cause analysis.

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

# Simulate reference and production with drift in some features
np.random.seed(42)
n_features = 5
reference = pd.DataFrame(
    np.random.normal(0, 1, (5000, n_features)),
    columns=[f'feature_{i}' for i in range(n_features)]
)
# Introduce drift in features 1 and 3
production = pd.DataFrame(
    np.random.normal([0, 1.5, 0, 2.0, 0], 1, (1000, n_features)),
    columns=reference.columns
)

drift_report = {}
for col in reference.columns:
    stat, p_val = ks_2samp(reference[col], production[col])
    drift_report[col] = {'ks_stat': round(stat, 4), 'p_value': round(p_val, 4),
                          'drift': p_val < 0.05}

for feature, result in drift_report.items():
    status = '*** DRIFT ***' if result['drift'] else 'stable'
    print(f'{feature}: KS={result["ks_stat"]} {status}')

Setting Alert Thresholds

Alert thresholds should be calibrated on historical data, not chosen arbitrarily. A common approach: compute the drift metric between your training set and multiple held-out validation splits to establish a baseline distribution of scores under no-drift conditions. Set the alert threshold at the 99th percentile of this baseline — any production score above this level is statistically unlikely to arise from the same distribution, indicating real drift.

import numpy as np
from scipy.stats import ks_2samp

train = np.random.normal(0, 1, 5000)

# Calibrate: compute KS statistic between train and 100 random validation splits
calibration_scores = []
for _ in range(100):
    val_sample = np.random.normal(0, 1, 500)  # same distribution
    stat, _ = ks_2samp(train, val_sample)
    calibration_scores.append(stat)

threshold_99 = np.percentile(calibration_scores, 99)
print(f'No-drift KS scores -- mean: {np.mean(calibration_scores):.4f}')
print(f'99th percentile threshold: {threshold_99:.4f}')

# Any production score above this triggers an alert
prod_sample_drifted = np.random.normal(0.5, 1, 500)  # shifted
stat, _ = ks_2samp(train, prod_sample_drifted)
print(f'Production KS: {stat:.4f} -- ALERT: {stat > threshold_99}')

Drift in Categorical Features

For categorical features, drift means the frequency distribution of categories changes. A country field might shift from 80% USA in training to 60% USA in production as the product expands globally. Use Chi-squared tests or Jensen-Shannon divergence to compare categorical distributions. New categories appearing in production that were absent from training are a special case — they result in KeyError in label encoders that were not fitted with handle_unknown='ignore'.

import numpy as np
from scipy.stats import chi2_contingency

# Training distribution
train_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.7, 0.15, 0.1, 0.05], size=5000
)
# Production: more EU traffic
prod_countries = np.random.choice(
    ['US', 'GB', 'DE', 'FR'], p=[0.5, 0.2, 0.2, 0.1], size=1000
)

categories = ['US', 'GB', 'DE', 'FR']
train_counts = [np.sum(train_countries == c) for c in categories]
prod_counts = [np.sum(prod_countries == c) for c in categories]

contingency = np.array([train_counts, prod_counts])
chi2, p_value, dof, _ = chi2_contingency(contingency)
print(f'Chi-squared: {chi2:.2f}, p-value: {p_value:.4f}')
print('Categorical drift detected:', p_value < 0.05)

Logging Drift Metrics Over Time

Drift monitoring is only useful if you log metrics consistently over time and visualise trends. Store weekly drift scores in a time-series database (InfluxDB, PostgreSQL, or even a CSV file) and plot them with a rolling window. A sudden spike in drift score pinpoints the exact week that a data pipeline changed or an external event shifted user behaviour, helping the team perform faster root-cause analysis.

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

records = []
base_date = datetime(2024, 1, 1)

for week in range(12):
    prod_sample = np.random.normal(week * 0.1, 1, 500)  # gradual drift
    from scipy.stats import ks_2samp
    train_sample = np.random.normal(0, 1, 5000)
    stat, p = ks_2samp(train_sample, prod_sample)
    records.append({
        'date': base_date + timedelta(weeks=week),
        'ks_stat': round(stat, 4),
        'p_value': round(p, 4),
        'alert': p < 0.05
    })

df = pd.DataFrame(records)
print(df.to_string(index=False))
# In production: df.to_sql('drift_log', engine, if_exists='append')

What to Do When Drift Is Detected

When drift is detected, there are several response strategies depending on severity. Minor drift: increase monitoring frequency and investigate the cause. Moderate drift: trigger retraining with the most recent data window. Severe drift: consider whether the old feature engineering is still valid for the new distribution, potentially requiring feature redesign. Always document the drift event and the business context (e.g., a marketing campaign, a platform change) that caused it.

def respond_to_drift(psi_score, ks_p_value):
    if psi_score < 0.1 and ks_p_value > 0.05:
        return 'No action needed. All features stable.'

    elif psi_score < 0.2 and ks_p_value > 0.01:
        return ('Moderate drift detected. '
                'Increase monitoring to daily. '
                'Schedule retraining for next cycle.')

    else:
        return ('SEVERE DRIFT. '
                'Trigger emergency retraining now. '
                'Consider feature engineering review. '
                'Alert data engineering team.')

# Example
print(respond_to_drift(0.05, 0.3))   # stable
print(respond_to_drift(0.15, 0.04))  # moderate
print(respond_to_drift(0.35, 0.001)) # severe

Visualising Feature Drift Over Time

Visualising drift as a time series helps stakeholders understand severity and timing. Plot the PSI or KS statistic for each monitored feature on a rolling weekly basis. When a feature line crosses the alert threshold, annotate the chart with the date and the nature of the business event that caused the shift (e.g., a marketing campaign, a pricing change, or a seasonal pattern). This converts raw statistics into actionable business intelligence.

import matplotlib.pyplot as plt
import numpy as np

weeks = list(range(1, 13))
psi_feature_A = [0.01, 0.02, 0.03, 0.05, 0.07, 0.11, 0.18, 0.22, 0.25, 0.24, 0.23, 0.26]
psi_feature_B = [0.01, 0.01, 0.02, 0.01, 0.02, 0.03, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08]

plt.figure(figsize=(10, 4))
plt.plot(weeks, psi_feature_A, marker='o', label='amount_usd (drifting)')
plt.plot(weeks, psi_feature_B, marker='s', label='merchant_category (stable)')
plt.axhline(0.1, color='orange', linestyle='--', label='Mild drift threshold')
plt.axhline(0.25, color='red', linestyle='--', label='Severe drift threshold')
plt.annotate('Pricing change', xy=(6, 0.11), xytext=(6, 0.16), arrowprops=dict(arrowstyle='->'))
plt.xlabel('Week')
plt.ylabel('PSI')
plt.legend()
plt.title('PSI Trend — Feature-Level Drift Monitoring')
plt.tight_layout()
plt.savefig('psi_trend.png', dpi=150)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: data drift is the shift in input feature distributions between training time and production time, causing silent model degradation, PSI and KS tests provide quantitative drift scores with industry-standard alert thresholds (PSI > 0.2 = major drift), and drift monitoring should cover all input features and log scores over time to enable trend analysis and root-cause identification. Next up we distinguish concept drift — when the relationship between inputs and the target label itself changes, which is a more subtle and dangerous phenomenon.

Pertanyaan yang Sering Diajukan

Apakah pelajaran “Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu” gratis?

Ya — teks lengkap “Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Apa yang akan aku pelajari di “Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu”?

Peserta akan mensimulasikan pergeseran dengan mengubah fitur input secara bertahap, menghitung Population Stability Index (PSI) dan divergensi KL, serta menetapkan peringatan berbasis ambang. Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.

Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?

Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 1 dari 4.

Berapa lama pelajaran “Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu” memakan waktu?

Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.

Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?

Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.

Semua pelajaran dalam kursus ini

  1. Pergeseran Data: Perubahan Distribusi Fitur Seiring Waktu
  2. Pergeseran Konsep: Ketika Hubungan antara X dan Y Berubah
  3. Memantau Distribusi Prediksi dan Skor Keyakinan
  4. Membangun Pipeline Peringatan Pergeseran dengan Evidently AI
← Kembali ke Machine Learning Academy