مراقبة التنبؤات: تسجيل المدخلات والمخرجات
سيضيف المتعلمون تسجيل التنبؤات إلى واجهة API، ويناقشون انحراف البيانات وتقادم النموذج، ويرسمون تصورًا لمشغّل إعادة التدريب وخط نشر النموذج.
مراقبة التنبؤات: تسجيل المدخلات والمخرجات درس مجاني في Machine Learning Academy على CoddyKit. هذا هو الدرس 4 من أصل 4. يمكنك قراءة الدرس كاملاً أدناه مجاناً — ثم تمرن عليه مباشرة في المتصفح باستخدام محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7. هذا الدرس جزء من مسار التعلم في Machine Learning Academy، وتقدمك يتزامن عبر الويب وتطبيق CoddyKit. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.
بعض أجزاء هذا الدرس لم تُترجم بعد وتظهر باللغة الإنجليزية.
Why Monitor a Deployed Model?
A model that performed excellently at deployment can silently degrade as the real world changes. Customer behaviour shifts, new product categories appear, sensors drift, economic conditions change — all causing the input distribution to diverge from training data. Without monitoring, you discover model failure when users complain or business metrics drop. Prediction monitoring catches degradation early, before it impacts users.
What to Log: The Three W's
Every prediction event should log: When (timestamp), What (input features and the model's output/prediction/probability), and ideally Whether (the eventual ground truth label, once available). The input features are needed for data drift detection; the outputs enable confidence monitoring; the ground truth enables accuracy monitoring over time.
Adding Logging to the FastAPI Endpoint
Extend the FastAPI prediction route to log each request to a structured JSONL file (JSON Lines — one JSON object per line). JSONL is easy to append to, easy to parse, and works well with tools like pandas, Spark, and cloud log aggregators. Each log line is one prediction event.
import json
from datetime import datetime
from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
app = FastAPI()
LOG_FILE = '/tmp/prediction_log.jsonl'
def log_prediction(features: dict, prediction: int, confidence: float):
entry = {
'timestamp': datetime.utcnow().isoformat(),
'features': features,
'predicted_class': prediction,
'confidence': confidence
}
with open(LOG_FILE, 'a') as f:
f.write(json.dumps(entry) + '\n')
# Call inside the predict route:
# log_prediction(features.dict(), pred, confidence)
print('Logging enabled — entries written to', LOG_FILE)Simulating and Reading the Log
After accumulating predictions, read the JSONL file into a pandas DataFrame for analysis. Each row is a prediction event. You can compute statistics over any time window: rolling mean confidence, feature distribution shifts, and prediction class frequency.
import json
import pandas as pd
from datetime import datetime
import random
# Simulate log entries
LOG_FILE = '/tmp/prediction_log.jsonl'
with open(LOG_FILE, 'w') as f:
for i in range(100):
entry = {
'timestamp': datetime.utcnow().isoformat(),
'features': {'sepal_length': round(4.5 + random.gauss(1, 0.5), 2),
'sepal_width': round(3.0 + random.gauss(0, 0.3), 2),
'petal_length': round(1.0 + random.gauss(2, 1), 2),
'petal_width': round(0.2 + random.gauss(0.5, 0.2), 2)},
'predicted_class': random.choice([0, 1, 2]),
'confidence': round(random.uniform(0.6, 1.0), 4)
}
f.write(json.dumps(entry) + '\n')
# Read back as DataFrame
rows = [json.loads(line) for line in open(LOG_FILE)]
df = pd.json_normalize(rows)
print(df.shape)
print(df[['confidence', 'predicted_class']].describe())Monitoring Confidence Over Time
Plot rolling mean confidence over time. A downward trend signals that the model is becoming less certain about its predictions — a classic early warning sign of data drift. Set a threshold (e.g., rolling confidence below 0.7) and trigger an alert. This gives you a warning before accuracy actually drops.
import pandas as pd
import matplotlib.pyplot as plt
import json
df = pd.json_normalize([json.loads(l) for l in open('/tmp/prediction_log.jsonl')])
df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.sort_values('timestamp').reset_index(drop=True)
# Rolling window of 20 predictions
df['rolling_confidence'] = df['confidence'].rolling(20).mean()
plt.figure(figsize=(10, 4))
plt.plot(df.index, df['confidence'], alpha=0.3, label='Per-prediction')
plt.plot(df.index, df['rolling_confidence'], linewidth=2, label='Rolling mean (n=20)')
plt.axhline(0.7, color='red', linestyle='--', label='Alert threshold')
plt.xlabel('Prediction number')
plt.ylabel('Confidence')
plt.title('Prediction Confidence Over Time')
plt.legend()
plt.show()Detecting Feature Distribution Shifts
Compare the distribution of each input feature in recent predictions against the training distribution. The Population Stability Index (PSI) is a common metric: PSI < 0.1 means no significant shift; 0.1–0.2 is moderate drift; >0.2 is severe drift requiring retraining. Alternatively, a Kolmogorov-Smirnov test tests whether two samples come from the same distribution.
import numpy as np
from scipy import stats
import pandas as pd
import json
df = pd.json_normalize([json.loads(l) for l in open('/tmp/prediction_log.jsonl')])
# Simulated training distribution for sepal_length
train_sepal = np.random.normal(5.8, 0.8, 500) # reference
recent_sepal = df['features.sepal_length'].values
# KS test: tests if two samples come from same distribution
stat, p_value = stats.ks_2samp(train_sepal, recent_sepal)
print(f'KS statistic: {stat:.4f} p-value: {p_value:.4f}')
if p_value < 0.05:
print('WARNING: sepal_length distribution has drifted!')
else:
print('No significant drift detected in sepal_length')Monitoring Prediction Class Frequencies
Plot the rolling frequency of each predicted class. If class 0 was predicted 60% of the time during training-period evaluation but is now being predicted only 10% of the time, something in the input data has shifted. Class distribution shifts are often easier to detect than subtle feature distribution changes and can serve as a quick first alert.
import pandas as pd
import json
df = pd.json_normalize([json.loads(l) for l in open('/tmp/prediction_log.jsonl')])
# Compute rolling class frequencies in windows of 20
window = 20
rolling_freq = pd.get_dummies(df['predicted_class']).rolling(window).mean()
rolling_freq.columns = [f'class_{c}' for c in rolling_freq.columns]
print('Recent class frequencies (last 20 predictions):')
print(rolling_freq.tail(1).to_string())
print('\nTraining class frequency (expected):')
print('class_0: 0.33 class_1: 0.33 class_2: 0.33')Structured Logging for Production
In production, write logs to a structured logging system rather than a flat file. Options include: Python logging module with JSON formatter, cloud logging services (AWS CloudWatch, GCP Cloud Logging, Datadog), or a database. Structured logs are queryable, scalable, and integrate with dashboards like Grafana for real-time monitoring.
import logging
import json
from datetime import datetime
# Configure structured JSON logging
class JSONFormatter(logging.Formatter):
def format(self, record):
log_data = {
'time': datetime.utcnow().isoformat(),
'level': record.levelname,
'message': record.getMessage()
}
if hasattr(record, 'prediction_data'):
log_data.update(record.prediction_data)
return json.dumps(log_data)
logger = logging.getLogger('ml_predictor')
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)
# Usage in prediction route:
extra = {'prediction_data': {'features': {'x': 1.5}, 'pred': 0, 'conf': 0.95}}
logger.info('prediction', extra=extra)Data Drift vs Concept Drift
Two distinct failure modes exist: Data drift — the distribution of input features changes (e.g., a sensor recalibrates, new customer demographics emerge). Concept drift — the relationship between inputs and the target changes (e.g., what constitutes 'fraud' evolves as fraudsters adapt). Data drift can sometimes be handled by retraining on fresh data; concept drift requires rethinking features or the modelling approach.
Retraining Triggers
Define clear retraining triggers before deployment. Common triggers: scheduled (retrain every 30 days regardless), performance-based (retrain if AUC on a labelled holdout drops below a threshold), or drift-based (retrain when PSI exceeds 0.2 on any key feature). Automating the trigger and the retraining pipeline prevents the model from silently degrading.
from datetime import datetime, timedelta
class RetrainingPolicy:
def __init__(self, last_trained: datetime, max_age_days=30,
min_auc=0.85, max_drift_psi=0.2):
self.last_trained = last_trained
self.max_age_days = max_age_days
self.min_auc = min_auc
self.max_drift_psi = max_drift_psi
def should_retrain(self, current_auc=None, max_psi=None):
age = (datetime.utcnow() - self.last_trained).days
if age >= self.max_age_days:
return True, f'Model is {age} days old (max {self.max_age_days})'
if current_auc and current_auc < self.min_auc:
return True, f'AUC {current_auc:.3f} below threshold {self.min_auc}'
if max_psi and max_psi > self.max_drift_psi:
return True, f'PSI {max_psi:.3f} above threshold {self.max_drift_psi}'
return False, 'No trigger met — model healthy'
policy = RetrainingPolicy(last_trained=datetime.utcnow() - timedelta(days=35))
print(policy.should_retrain(current_auc=0.90))The Monitoring Dashboard Sketch
A production ML monitoring dashboard typically shows: rolling prediction confidence over time, class distribution over time, feature distribution histograms comparing training vs recent predictions, and (when labels are available) rolling accuracy or AUC. Tools like Grafana + Prometheus, Evidently AI, or WhyLogs provide pre-built components for ML monitoring without building from scratch.
# Evidently AI example (pip install evidently)
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
import pandas as pd
import numpy as np
# Reference: training feature distributions
reference = pd.DataFrame({
'sepal_length': np.random.normal(5.8, 0.8, 300),
'petal_length': np.random.normal(3.7, 1.7, 300)
})
# Current: recent predictions features
current = pd.DataFrame({
'sepal_length': np.random.normal(6.5, 1.2, 100), # drifted!
'petal_length': np.random.normal(3.7, 1.7, 100)
})
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference, current_data=current)
report.save_html('/tmp/drift_report.html')
print('Drift report saved to /tmp/drift_report.html')Quick Check
Test your understanding of prediction monitoring and logging from this lesson.
Lesson Recap
In this lesson you learned: log every prediction event with timestamp, input features, predicted class, and confidence to enable drift detection, monitor rolling confidence and feature distributions to catch degradation before it affects users, and define automated retraining triggers (schedule, performance threshold, or drift threshold) to keep the model healthy in production. This completes the Model Persistence and Deployment Basics course.
تعلم Python مع معلم ذكاء اصطناعي — مجانًا
اكتب وقم بتشغيل أكوادك الفعلية في المتصفح، واحصل على مساعدة فورية من معلم ذكاء اصطناعي متاح 24/7، واستمر من حيث توقفت على الويب أو في التطبيق.
- الدورات
- 30
- الدروس
- 120
الأسئلة الشائعة
هل درس «مراقبة التنبؤات: تسجيل المدخلات والمخرجات» مجاني؟
نعم — نص درس «مراقبة التنبؤات: تسجيل المدخلات والمخرجات» كامل متاح مجاناً هنا على الويب. لتمرينه بشكل تفاعلي (محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7) وفتح باقي دورة Machine Learning Academy، انتقل إلى CoddyKit PRO. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.
ماذا ستتعلم في «مراقبة التنبؤات: تسجيل المدخلات والمخرجات»؟
سيضيف المتعلمون تسجيل التنبؤات إلى واجهة API، ويناقشون انحراف البيانات وتقادم النموذج، ويرسمون تصورًا لمشغّل إعادة التدريب وخط نشر النموذج. تتمرن على Machine Learning Academy مع أكواد عملية تشغلها مباشرة في المتصفح، ومدرس ذكاء اصطناعي متاح 24/7 يجيب على أسئلتك أثناء عملك.
هل أحتاج إلى خبرة سابقة لأبدأ Machine Learning Academy؟
لا تُشترط خبرة سابقة. Machine Learning Academy على CoddyKit منظم للمبتدئين حتى المتقدمين، لذا يمكنك البدء من هنا أو من البداية والتقدم بسرعتك الخاصة. هذا هو الدرس 4 من أصل 4.
كم من الوقت يستغرق درس «مراقبة التنبؤات: تسجيل المدخلات والمخرجات»؟
معظم دروس CoddyKit تستغرق حوالي 5–10 دقائق. كل منها موجز وتفاعلي، لذا تحرز تقدماً مستمراً وتستأنف من حيث توقفت عبر الويب والتطبيق.
هل يمكنني كتابة وتشغيل أكواد في درس Machine Learning Academy هذا؟
نعم. كل درس في Machine Learning Academy يتضمن محرر أكواد مدمج، لذا تكتب وتشغل أكواداً حقيقية مباشرة في متصفحك وتحصل على تعليقات فورية من الذكاء الاصطناعي — بدون إعداد محلي.
جميع الدروس في هذه الدورة
- حفظ النماذج باستخدام joblib وpickle
- إدارة إصدارات النماذج: أهمية أسماء الملفات والبيانات الوصفية
- تقديم التنبؤات عبر نقطة نهاية FastAPI
- مراقبة التنبؤات: تسجيل المدخلات والمخرجات