Monitoring Model Performance in Production
Data drift detection, model degradation signals, retraining triggers, evidently library.
Monitoring Model Performance in Production is a free Learn AI with Python lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Monitor in Production?
A model that performed well at launch can silently degrade as the world changes. Monitoring detects this drift early, before it quietly tanks business metrics, so you know when to retrain.
Data Drift vs Concept Drift
- Data drift: the input distribution changes (new customer mix).
- Concept drift: the relationship between inputs and target changes (what predicts churn shifts).
Both hurt accuracy; monitoring catches them in different ways.
PSI for Data Drift
The Population Stability Index (PSI) compares a features distribution now versus a baseline. Bin both, then sum (now% - base%) * ln(now% / base%) across bins.
Reading PSI Values
Rules of thumb: PSI < 0.1 no significant shift, 0.1-0.25 moderate shift (investigate), > 0.25 major shift (likely retrain). PSI is computed per feature.
import numpy as np
def psi(base, now, bins=10):
edges = np.percentile(base, np.linspace(0, 100, bins + 1))
b_pct = np.histogram(base, edges)[0] / len(base) + 1e-6
n_pct = np.histogram(now, edges)[0] / len(now) + 1e-6
return np.sum((n_pct - b_pct) * np.log(n_pct / b_pct))KS Test for Distributions
The Kolmogorov-Smirnov (KS) test compares two continuous distributions. A small p-value means the production distribution differs significantly from the training distribution, signaling drift.
from scipy.stats import ks_2samp
stat, pvalue = ks_2samp(train_feature, prod_feature)
if pvalue < 0.05:
print("Distribution drift detected")PSI vs KS
- PSI: single interpretable number, great for dashboards and thresholds.
- KS: a statistical hypothesis test with a p-value, more rigorous for continuous features.
Many teams report both side by side.
What Is Evidently?
Evidently is an open-source library that automates drift and quality reports. Instead of coding every metric by hand, you assemble preset checks and get an HTML/JSON report.
# pip install evidently
from evidently.report import Report
from evidently.metric_preset import DataDriftPresetBuilding a DataDriftPreset Report
The DataDriftPreset runs a full suite of drift tests across all columns, comparing a reference (training) dataset to current (production) data.
report = Report(metrics=[DataDriftPreset()])
report.run(
reference_data=train_df,
current_data=prod_df
)
report.save_html("drift_report.html")Extracting Results Programmatically
Export the report as a dict to feed alerting logic, so monitoring runs unattended on a schedule.
result = report.as_dict()
drift_share = result["metrics"][0]["result"]["dataset_drift"]
print("Dataset drift flag:", drift_share)Alerting on Accuracy Drop
When you have ground-truth labels, track live accuracy and fire an alert when it falls below a threshold, the most direct signal that retraining is overdue.
THRESHOLD = 0.85
if live_accuracy < THRESHOLD:
send_alert(f"Accuracy dropped to {live_accuracy:.2f}")Closing the Loop
Good monitoring connects to action: detect drift -> alert -> trigger a retraining pipeline (your Makefile/registry from earlier lessons) -> deploy the new version. Monitoring without a response plan is just noise.
Quick Check
Test your monitoring knowledge.
Recap
You monitored production models: PSI and the KS test detect data drift, Evidently automates reports via DataDriftPreset, and you alert when accuracy drops below a threshold, then trigger retraining. That ends the MLOps course. Next course: Serving AI Models with FastAPI.
Frequently asked questions
Is the “Monitoring Model Performance in Production” lesson free?
Yes — the full text of “Monitoring Model Performance in Production” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Monitoring Model Performance in Production”?
Data drift detection, model degradation signals, retraining triggers, evidently library. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Monitoring Model Performance in Production” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Experiment Tracking with MLflow
- Model Registry and Versioning
- Building Reproducible ML Pipelines
- Monitoring Model Performance in Production