AI Prompt Engineering · Lesson

Monitoring Prompt Performance in Production

Tracking latency, cost, quality scores, and failure rates per prompt version.

Monitoring Prompt Performance in Production is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Monitor Prompt Performance?

A prompt deployed to production is not 'done'. Model behavior drifts with updates, user inputs change over time, and cost can spike unexpectedly. Continuous monitoring detects regressions before they hurt users and keeps spending predictable.

Key Metrics per Prompt Version

Track these five metrics per prompt version in production:

Avg latency: time from request to full response (P50, P95, P99)
Cost per call: input tokens × price + output tokens × price
Quality score: automated eval (LLM-as-judge or task metric)
Error rate: API errors + malformed output parse failures
User satisfaction: thumbs rating, retry rate, session abandonment

Instrumentation: Recording Metrics

Wrap every LLM call with instrumentation that records all key metrics to a time-series store or database for later analysis and alerting.

import time
import openai

client = openai.OpenAI(api_key='YOUR_API_KEY')

def instrumented_call(prompt_id, version, messages, model):
    start = time.time()
    error = False
    response = None
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages
        )
    except Exception as e:
        error = True
        raise
    finally:
        latency_ms = (time.time() - start) * 1000
        usage = response.usage if response else None
        record_metric({
            'prompt_id': prompt_id,
            'version': version,
            'latency_ms': latency_ms,
            'input_tokens': usage.prompt_tokens if usage else 0,
            'output_tokens': usage.completion_tokens if usage else 0,
            'error': error,
            'timestamp': time.time()
        })
    return response

Calculating Cost per Call

Cost per call = input tokens × input price + output tokens × output price. Storing raw token counts lets you recalculate cost whenever pricing changes.

# pricing.py — model pricing table (per 1M tokens)
PRICING = {
    'gpt-4o': {'input': 2.50, 'output': 10.00},
    'gpt-4o-mini': {'input': 0.15, 'output': 0.60},
    'claude-3-5-sonnet-20241022': {'input': 3.00, 'output': 15.00},
}

def cost_usd(model, input_tokens, output_tokens):
    p = PRICING.get(model)
    if not p:
        return None
    cost = (input_tokens / 1_000_000) * p['input'] \
         + (output_tokens / 1_000_000) * p['output']
    return round(cost, 6)

# Example
c = cost_usd('gpt-4o-mini', input_tokens=800, output_tokens=200)
print(f'Cost per call: ${c}')   # Cost per call: $0.000240

# Daily cost estimate
calls_per_day = 50_000
print(f'Daily cost: ${round(c * calls_per_day, 2)}')  # $12.00

Automated Quality Scoring

Use an LLM judge to score output quality automatically at production scale. Sample 5-10% of calls for quality scoring to keep costs manageable.

import random

JUDGE_SAMPLE_RATE = 0.05  # score 5% of calls

JUDGE_PROMPT = '''Rate the quality of this AI response on a scale of 1-5.
1=Very poor, 3=Acceptable, 5=Excellent.
Return only the integer score.

User input: {input}
AI response: {output}'''

def maybe_score_quality(user_input, ai_output, model='gpt-4o-mini'):
    if random.random() > JUDGE_SAMPLE_RATE:
        return None  # not sampled

    judge_messages = [{'role': 'user', 'content':
        JUDGE_PROMPT.format(input=user_input, output=ai_output)}]
    response = client.chat.completions.create(
        model=model, messages=judge_messages, max_tokens=5
    )
    try:
        score = int(response.choices[0].message.content.strip())
        return max(1, min(5, score))  # clamp to 1-5
    except ValueError:
        return None

Storing Metrics in a Time-Series DB

Time-series databases (InfluxDB, TimescaleDB, or even a simple PostgreSQL table with a timestamp index) store per-call metrics efficiently and support aggregation queries for dashboards.

-- TimescaleDB / PostgreSQL schema for prompt metrics
CREATE TABLE prompt_metrics (
  ts              TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  prompt_id       VARCHAR(100),
  version         VARCHAR(20),
  model           VARCHAR(50),
  latency_ms      FLOAT,
  input_tokens    INT,
  output_tokens   INT,
  cost_usd        FLOAT,
  quality_score   FLOAT,   -- NULL if not sampled
  error           BOOLEAN DEFAULT FALSE
);

-- Convert to hypertable (TimescaleDB)
SELECT create_hypertable('prompt_metrics', 'ts');

-- Query: hourly P95 latency by version
SELECT
  date_trunc('hour', ts) AS hour,
  version,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95_latency
FROM prompt_metrics
WHERE prompt_id = 'summarize-article'
  AND ts > NOW() - INTERVAL '24 hours'
GROUP BY 1, 2
ORDER BY 1;

Building a Metrics Dashboard

Dashboards surface the five key metrics in real time. Use Grafana (with TimescaleDB datasource) or a custom web dashboard. Key panels:

Latency P50/P95/P99 over time, by version
Cost per call trend (daily/weekly)
Error rate percentage
Quality score rolling average
User satisfaction score

# Simple dashboard query aggregation in Python
def get_dashboard_stats(conn, prompt_id, hours=24):
    with conn.cursor() as cur:
        cur.execute('''
            SELECT
              version,
              COUNT(*) AS calls,
              AVG(latency_ms) AS avg_latency,
              PERCENTILE_CONT(0.95)
                WITHIN GROUP (ORDER BY latency_ms) AS p95_latency,
              AVG(cost_usd) AS avg_cost,
              AVG(quality_score) FILTER (WHERE quality_score IS NOT NULL) AS avg_quality,
              SUM(error::int)::float / COUNT(*) AS error_rate
            FROM prompt_metrics
            WHERE prompt_id = %s
              AND ts > NOW() - INTERVAL %s
            GROUP BY version
            ORDER BY MAX(ts) DESC
        ''', (prompt_id, f'{hours} hours'))
        return cur.fetchall()

Anomaly Detection for Prompt Regressions

Manual dashboard review misses slow regressions. Automated anomaly detection compares a rolling window against a historical baseline and fires an alert when deviation exceeds a threshold.

import statistics

def detect_anomaly(recent_values, baseline_values, threshold_std=2.0):
    if len(baseline_values) < 10:
        return False  # not enough data
    baseline_mean = statistics.mean(baseline_values)
    baseline_std = statistics.stdev(baseline_values)
    recent_mean = statistics.mean(recent_values)

    if baseline_std == 0:
        return False
    z_score = abs(recent_mean - baseline_mean) / baseline_std
    return z_score > threshold_std

# Example: detect quality regression
baseline_quality = [4.1, 4.0, 4.2, 4.1, 4.0, 3.9, 4.1, 4.2, 4.0, 4.1]
recent_quality   = [3.2, 3.1, 3.4, 3.0]

if detect_anomaly(recent_quality, baseline_quality):
    print('ALERT: Quality score anomaly detected!')
    # fire_pagerduty_alert(...)
else:
    print('Quality within normal range')

Alerting Rules and Thresholds

Define alert rules as code so they are version-controlled and reviewable. Common prompt monitoring alerts:

Error rate > 5% for 5 consecutive minutes
P95 latency > 10s for 3 minutes
Cost per day > 2× weekly average (cost spike)
Quality score drops > 20% from baseline

# alerts.yaml (Grafana alerting or custom)
alerts:
  - name: HighErrorRate
    condition: error_rate > 0.05
    for: 5m
    severity: critical
    message: 'Prompt {prompt_id} error rate {error_rate:.1%} exceeds 5%'

  - name: HighLatencyP95
    condition: p95_latency_ms > 10000
    for: 3m
    severity: warning
    message: 'P95 latency {p95_latency_ms}ms exceeds 10s for {prompt_id}'

  - name: CostSpike
    condition: daily_cost_usd > weekly_avg_daily_cost * 2
    for: 1h
    severity: warning
    message: 'Daily cost ${daily_cost_usd:.2f} is 2x the weekly average'

  - name: QualityRegression
    condition: rolling_quality_avg < baseline_quality_avg * 0.80
    for: 15m
    severity: critical
    message: 'Quality dropped {pct_drop:.0%} for prompt {prompt_id}'

User Satisfaction Signals

Automated metrics do not capture everything. Collect explicit and implicit user signals to complement quality scores:

Thumbs rating: explicit 👍/👎 on AI responses
Retry rate: user immediately re-submits same query (implicit dissatisfaction)
Copy/use rate: user copies AI output (implicit satisfaction)
Correction rate: user edits AI output before using it

# feedback_collector.py
def record_feedback(prompt_id, version, call_id, signal_type, value):
    '''
    signal_type: 'thumbs' | 'retry' | 'copy' | 'edit'
    value: for thumbs: 1 (up) or -1 (down); others: 1 (occurred)
    '''
    db.execute(
        'INSERT INTO prompt_feedback '
        '(prompt_id, version, call_id, signal_type, value, ts) '
        'VALUES (%s, %s, %s, %s, %s, NOW())',
        (prompt_id, version, call_id, signal_type, value)
    )

# Aggregate satisfaction score
def satisfaction_score(prompt_id, version):
    rows = db.fetch(
        'SELECT signal_type, AVG(value) as avg_val FROM prompt_feedback '
        'WHERE prompt_id=%s AND version=%s GROUP BY signal_type',
        (prompt_id, version)
    )
    return {r['signal_type']: round(r['avg_val'], 3) for r in rows}

Version Comparison Reports

When evaluating whether to promote or rollback a canary, generate a side-by-side comparison of all metrics between the new version and the baseline. This report drives the promotion decision.

def version_comparison_report(prompt_id, version_a, version_b, hours=48):
    stats_a = get_dashboard_stats_for_version(prompt_id, version_a, hours)
    stats_b = get_dashboard_stats_for_version(prompt_id, version_b, hours)

    print(f'=== Comparison: {version_a} vs {version_b} ===')
    metrics = ['avg_latency', 'p95_latency', 'avg_cost', 'avg_quality', 'error_rate']
    for m in metrics:
        va = stats_a.get(m, 0)
        vb = stats_b.get(m, 0)
        delta = vb - va
        pct = (delta / va * 100) if va else 0
        direction = '+' if delta > 0 else ''
        print(f'{m:20s}: {va:.4f} -> {vb:.4f}  ({direction}{pct:.1f}%)')

# Example output:
# avg_latency         : 1234.5000 -> 1189.2000  (-3.7%)
# p95_latency         : 3210.0000 -> 3050.0000  (-5.0%)
# avg_cost            : 0.000240 -> 0.000255  (+6.2%)
# avg_quality         : 4.1000 -> 4.3000  (+4.9%)
# error_rate          : 0.0120 -> 0.0080  (-33.3%)

Quick Check

You want to detect quality regressions automatically without manually checking dashboards. Which approach best accomplishes this?

Monitoring Summary

Production prompt monitoring requires tracking five dimensions per version:

Latency (P50/P95/P99) — user experience
Cost per call — financial health
Quality score — automated LLM-judge sampling
Error rate — reliability
User satisfaction — thumbs rating, retry, copy signals

Store metrics in a time-series database, visualize in dashboards, and fire alerts when thresholds are breached. Version comparison reports inform promotion and rollback decisions.

Frequently asked questions

Is the “Monitoring Prompt Performance in Production” lesson free?

Yes — the full text of “Monitoring Prompt Performance in Production” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Monitoring Prompt Performance in Production”?

Tracking latency, cost, quality scores, and failure rates per prompt version. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Monitoring Prompt Performance in Production” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Prompt Registry Architecture
Version Control for Prompts
Deployment and Rollback Strategies
Monitoring Prompt Performance in Production

← Back to AI Prompt Engineering