Concept Drift: When the Relationship Between X and Y Changes
Learners will distinguish data drift from concept drift using a time-series example, and understand why data drift does not always imply model performance degradation.
Concept Drift: When the Relationship Between X and Y Changes is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is Concept Drift?
Concept drift occurs when the relationship between input features X and the target label Y changes over time — even if the feature distributions themselves remain stable. In fraud detection, fraudsters learn which transactions are flagged and change their patterns, so transactions that looked fraudulent in 2022 look benign by 2024. The 'concept' being learned (what fraud looks like) has drifted, but the feature distributions may appear unchanged, making concept drift harder to detect than data drift.
Data Drift vs Concept Drift: Key Difference
The critical distinction: data drift is a change in P(X) — the input feature distribution. Concept drift is a change in P(Y|X) — the conditional relationship between features and labels. Data drift does not always degrade model performance (if the new distribution is still within the learned decision boundary). Concept drift always degrades performance because the model's learned mapping from X to Y is now incorrect, regardless of whether X looks the same.
import numpy as np
import pandas as pd
# Scenario: credit scoring model
# Feature: income. Label: 1 = defaults, 0 = repays
# Training period: incomes 30-50k correlate with defaults
np.random.seed(42)
train_income = np.random.normal(40000, 8000, 1000)
train_default = (train_income < 35000).astype(int) # low income -> default
# Concept drift: inflation shifts threshold; now defaults start at 50k
prod_income = np.random.normal(40000, 8000, 500) # SAME distribution (no data drift)
prod_default = (prod_income < 50000).astype(int) # new relationship
print('Train default rate:', train_default.mean().round(3))
print('Prod default rate:', prod_default.mean().round(3))
print('Feature distribution same (no data drift).',
'But Y|X relationship has changed (concept drift).')Types of Concept Drift
Concept drift comes in four varieties. Sudden drift: the relationship changes abruptly (e.g., COVID-19 changing consumer spending patterns overnight). Gradual drift: the old concept slowly fades and a new concept emerges (e.g., shifting definition of spam over months). Incremental drift: a continuous, slow drift with no clear breakpoint. Recurring drift: seasonal patterns — the relationship at Christmas differs from July, but July eventually returns.
import numpy as np
import matplotlib.pyplot as plt
t = np.linspace(0, 100, 1000)
# Sudden drift: step function at t=50
sudden = np.where(t < 50, 0.8, 0.3) # accuracy drops at t=50
# Gradual drift: linear transition
gradual = np.where(t < 40, 0.8,
np.where(t > 60, 0.4,
0.8 - (t - 40) * 0.02))
# Recurring drift: seasonal pattern
recurring = 0.6 + 0.2 * np.sin(2 * np.pi * t / 20)
for name, series in [('Sudden', sudden), ('Gradual', gradual), ('Recurring', recurring)]:
print(f'{name} drift -- min acc: {series.min():.2f}, max: {series.max():.2f}')Detecting Concept Drift via Performance Monitoring
The most reliable signal for concept drift is declining model performance on labelled production data. Track your chosen metric (accuracy, F1, AUC) on a rolling window of recent predictions where ground-truth labels have become available. A downward trend in performance that is not explained by data drift (the features look the same) is strong evidence of concept drift. This requires a feedback loop: collecting true labels for production predictions, which can take days to months depending on the task.
import numpy as np
import pandas as pd
np.random.seed(42)
# Simulate weekly model accuracy tracking
weekly_accuracy = [
0.92, 0.91, 0.90, 0.89, 0.88, # slight decline
0.86, 0.83, 0.79, 0.74, 0.68, # accelerating decline = concept drift
0.65, 0.61
]
df = pd.DataFrame({'week': range(1, 13), 'accuracy': weekly_accuracy})
df['rolling_mean'] = df['accuracy'].rolling(3).mean()
df['alert'] = df['accuracy'] < 0.75 # threshold
print(df.to_string(index=False))
print('Weeks with alerts:', df[df['alert']]['week'].tolist())Detecting Concept Drift Without Labels
Waiting for ground-truth labels is slow. When labels arrive with high latency, use prediction distribution monitoring as an early warning signal. If P(X) is stable (no data drift) but the model's predicted label distribution shifts significantly, this suggests the underlying concept has changed. For example, if a binary classifier's positive-prediction rate drops from 30% to 5% without any change in feature distributions, the model is likely seeing a different relationship between features and the positive class.
import numpy as np
# Monitoring prediction distributions as a proxy for concept drift
# Period 1 (training distribution): ~30% positive predictions
period1_probs = np.random.beta(2, 5, 1000) # right-skewed, ~30% above 0.5
period1_pos_rate = (period1_probs > 0.5).mean()
# Period 2 (concept drifted): only ~5% positive predictions (model confused)
period2_probs = np.random.beta(1, 15, 500) # very right-skewed
period2_pos_rate = (period2_probs > 0.5).mean()
print(f'Period 1 positive rate: {period1_pos_rate:.2%}')
print(f'Period 2 positive rate: {period2_pos_rate:.2%}')
print(f'Rate drop: {(period1_pos_rate - period2_pos_rate):.2%}')
print('Possible concept drift detected!')Windowed Retraining: Adapting to Concept Drift
The most common strategy for handling concept drift is windowed retraining: periodically retrain the model using only the most recent N examples, discarding old data that reflects outdated patterns. A sliding window keeps the most recent N examples; an expanding window uses all data with recency weighting. The optimal window size depends on how fast the concept evolves — fast-changing domains (financial markets) need smaller windows than slow-changing ones (weather forecasting).
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
np.random.seed(42)
def simulate_concept_drift(n, t):
X = np.random.normal(0, 1, (n, 2))
# Decision boundary rotates over time
y = ((X[:, 0] * np.cos(t) - X[:, 1] * np.sin(t)) > 0).astype(int)
return X, y
# Full retrain vs sliding window retrain
for window_size in [100, 500, None]: # None = all data
all_X, all_y = [], []
accs = []
for t in np.linspace(0, np.pi, 10):
X, y = simulate_concept_drift(200, t)
all_X.append(X)
all_y.append(y)
if window_size:
train_X = np.vstack(all_X[-window_size//200:])
train_y = np.hstack(all_y[-window_size//200:])
else:
train_X = np.vstack(all_X)
train_y = np.hstack(all_y)
clf = LogisticRegression().fit(train_X, train_y)
accs.append(clf.score(X, y))
print(f'Window={window_size}: avg accuracy={np.mean(accs):.3f}')Sample Weighting: Downweighting Old Data
Instead of a hard window cutoff, assign exponentially decaying weights to training examples so recent examples contribute more to the model. Examples from last week get weight 1.0; examples from last month get weight 0.5; examples from six months ago get weight 0.1. Many scikit-learn estimators accept a sample_weight parameter in their fit method, making this easy to implement without discarding any data.
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
np.random.seed(42)
# Simulate 1000 training examples collected over 10 weeks
X = np.random.normal(0, 1, (1000, 5))
y = (X[:, 0] + np.random.normal(0, 0.5, 1000) > 0).astype(int)
weeks_ago = np.linspace(10, 0, 1000) # example 0 is oldest, 999 is newest
# Exponential decay: half-life of 3 weeks
half_life = 3.0
sample_weights = np.exp(-weeks_ago * np.log(2) / half_life)
sample_weights /= sample_weights.sum()
clf = GradientBoostingClassifier(n_estimators=100, random_state=42)
clf.fit(X, y, sample_weight=sample_weights * len(y)) # scale up
print('Model trained with recency-weighted samples.')
print('Weight ratio new/old:', round(sample_weights[-1] / sample_weights[0], 2))CUSUM and ADWIN Drift Detectors
Dedicated concept drift detection algorithms provide more principled alerts. CUSUM (Cumulative Sum) detects changes in the mean of a sequence by accumulating deviations from a reference value. ADWIN (Adaptive Windowing) maintains a variable-length window and shrinks it when a statistical test detects a distribution change within the window. The river library (formerly scikit-multiflow) implements both for streaming data.
# Using the river library for streaming drift detection
# pip install river
# from river.drift import ADWIN
# adwin = ADWIN(delta=0.002) # delta controls sensitivity
# error_sequence = [0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1] # errors over time
#
# for i, error in enumerate(error_sequence):
# adwin.update(error)
# if adwin.drift_detected:
# print(f'ADWIN detected drift at step {i}')
# adwin = ADWIN(delta=0.002) # reset detector
# Manual CUSUM example:
def cusum(errors, threshold=5, drift=0.01):
cum_sum, alerts = 0, []
for i, e in enumerate(errors):
cum_sum = max(0, cum_sum + e - drift)
if cum_sum > threshold:
alerts.append(i)
cum_sum = 0
return alerts
errors = [0]*10 + [1]*15 # errors spike after step 10
print('CUSUM drift alerts at steps:', cusum(errors))Concept Drift vs Seasonality
Seasonality is predictable periodic variation (Christmas shopping spikes, summer travel patterns) while concept drift is an unpredictable one-directional shift. A good monitoring system distinguishes them by checking historical seasonality patterns. If the current distribution matches last year's same-period distribution, it is likely seasonality rather than drift. Include time-of-year features in your model to handle predictable seasonality rather than triggering unnecessary retraining.
import numpy as np
from scipy.stats import ks_2samp
np.random.seed(42)
# Training: July 2023 data
july_2023 = np.random.normal(100, 20, 1000) # low activity (summer)
# Monitoring: July 2024 (seasonal -- same month last year)
july_2024 = np.random.normal(102, 21, 500) # similar to last July
# Monitoring: December 2024 (seasonal spike -- same as Dec 2023)
dec_2024 = np.random.normal(180, 30, 500) # high activity (Christmas)
stat_july, p_july = ks_2samp(july_2023, july_2024)
stat_dec, p_dec = ks_2samp(july_2023, dec_2024)
print(f'July 2024 vs July 2023: KS={stat_july:.3f} p={p_july:.3f} -> {"drift" if p_july < 0.05 else "seasonal OK"}')
print(f'Dec 2024 vs July 2023: KS={stat_dec:.3f} p={p_dec:.3f} -> Expected seasonal shift')Building a Concept Drift Response Plan
Respond to concept drift with a tiered plan. Tier 1 (early warning): alert on prediction distribution shifts without waiting for labels. Tier 2 (confirmation): when labels arrive, confirm performance degradation exceeds 5% relative. Tier 3 (action): trigger automatic windowed retraining using the drift detection timestamp as the window cutoff. Tier 4 (escalation): if retraining does not recover performance, escalate to a data scientist for feature engineering review.
def concept_drift_response(current_accuracy, baseline_accuracy,
data_drift_detected, prediction_rate_shift):
relative_drop = (baseline_accuracy - current_accuracy) / baseline_accuracy
if relative_drop > 0.10:
return ('TIER 4 ESCALATION: >10% accuracy drop. '
'Manual feature engineering review required.')
elif relative_drop > 0.05:
return ('TIER 3 ACTION: 5-10% accuracy drop. '
'Trigger windowed retraining immediately.')
elif data_drift_detected or prediction_rate_shift > 0.15:
return ('TIER 2 MONITOR: Early warning signals present. '
'Increase monitoring to daily. Queue retraining.')
else:
return 'TIER 1 OK: No significant concept drift detected.'
print(concept_drift_response(0.84, 0.92, False, 0.05)) # TIER 3
print(concept_drift_response(0.91, 0.92, True, 0.18)) # TIER 2
print(concept_drift_response(0.91, 0.92, False, 0.03)) # TIER 1Ensemble Models and Concept Drift Resilience
Ensemble methods like random forests and gradient boosting can mask concept drift for longer than single models because their diversity makes them less sensitive to any one feature distribution shift. However, this resilience is a double-edged sword: the model degrades more slowly and more silently. Monitor the individual tree disagreement rate in a forest — when trees increasingly disagree on predictions, it is an early signal that the concept is drifting and the ensemble is struggling to reach consensus.
import numpy as np
from sklearn.ensemble import RandomForestClassifier
# Measure inter-tree disagreement on current data as drift signal
def inter_tree_disagreement(forest, X):
predictions = np.array([tree.predict(X) for tree in forest.estimators_])
# Fraction of pairs of trees that disagree per sample, averaged over samples
n_trees = len(forest.estimators_)
disagreement_per_sample = []
for i in range(X.shape[0]):
preds = predictions[:, i]
disagree = (preds != preds[0]).sum() / n_trees
disagreement_per_sample.append(disagree)
return np.mean(disagreement_per_sample)
# Higher disagreement = more concept drift
# print(inter_tree_disagreement(model, X_current))Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: concept drift is a change in P(Y|X) — the mapping from features to labels — which can occur even when feature distributions appear stable, declining model performance on labelled production data is the definitive signal of concept drift, and windowed retraining and sample weighting are the primary adaptation strategies to keep models current with evolving concepts. Next up we look at monitoring prediction distributions and confidence scores as an early warning system when ground-truth labels are not yet available.
Frequently asked questions
Is the “Concept Drift: When the Relationship Between X and Y Changes” lesson free?
Yes — the full text of “Concept Drift: When the Relationship Between X and Y Changes” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Concept Drift: When the Relationship Between X and Y Changes”?
Learners will distinguish data drift from concept drift using a time-series example, and understand why data drift does not always imply model performance degradation. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Concept Drift: When the Relationship Between X and Y Changes” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Data Drift: Feature Distribution Shifts Over Time
- Concept Drift: When the Relationship Between X and Y Changes
- Monitoring Prediction Distributions and Confidence Scores
- Building a Drift Alert Pipeline with Evidently AI