Measure Drift with PSI and KS
Quantify distribution shift on each feature.
Measure Drift with PSI and KS is a free MLOps Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the MLOps Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Hunch to Number
Saying a feature feels different is not enough. You need a number that says how much its distribution moved. Two trusted tools are PSI and the KS test. 📏
What PSI Measures
The Population Stability Index compares a feature now against training, bin by bin. The bigger the gap in each bin, the higher the score.
Reading PSI Values
A common rule of thumb: PSI below 0.1 means stable, 0.1 to 0.25 means moderate shift, and above 0.25 means significant drift worth acting on.
How PSI Is Computed
For each bin, PSI takes the percent difference times the log ratio of the two percentages, then sums across all bins into one score.
import numpy as np
def psi(expected, actual):
return np.sum((actual - expected) * np.log(actual / expected))Binning Comes First
PSI needs both samples placed into the same bins, often deciles of the training data. Same edges for both sides keep the comparison fair.
edges = np.quantile(reference, np.linspace(0, 1, 11))
exp = np.histogram(reference, edges)[0] / len(reference)
act = np.histogram(current, edges)[0] / len(current)What the KS Test Measures
The Kolmogorov-Smirnov test finds the largest gap between two cumulative distributions. That maximum distance is the KS statistic.
KS in One Line
SciPy gives you the KS test directly. A small p-value means the two samples likely come from different distributions, hinting at drift.
from scipy.stats import ks_2samp
stat, pvalue = ks_2samp(reference, current)
drifted = pvalue < 0.05PSI vs KS
PSI gives a single severity score great for dashboards. KS gives a statistical p-value great for yes-or-no decisions. Many teams track both side by side.
Categorical Features
PSI works naturally on categories, treating each value as its own bin. KS assumes ordered numbers, so reach for chi-square on categorical features instead.
Watch the Sample Size
With huge samples the KS test flags tiny, harmless shifts as significant. Pair its p-value with an effect size like PSI before you raise an alarm.
Per-Feature, Then Roll Up
Run these tests per feature, then summarize. A handy signal is the share of features that drifted, which tells you if the shift is broad or narrow. 📊
Quick Check
Pick the right read on these numbers.
Recap
You turned drift into numbers: PSI scores severity with bins, KS measures the biggest distribution gap. Use both, mind sample size, and pick the right test per feature type. 🎯
Frequently asked questions
Is the “Measure Drift with PSI and KS” lesson free?
Yes — the full text of “Measure Drift with PSI and KS” is free to read here on the web, and the MLOps Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the MLOps Academy course, upgrade to CoddyKit PRO.
What will I learn in “Measure Drift with PSI and KS”?
Quantify distribution shift on each feature. You practise MLOps Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start MLOps Academy?
No prior experience is required. MLOps Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Measure Drift with PSI and KS” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this MLOps Academy lesson?
Yes. Every MLOps Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Data Drift vs Concept Drift
- Measure Drift with PSI and KS
- Generate Drift Reports with Evidently
- Set Drift Thresholds and Triggers