Calibrating Judge Models Against Humans
Collect a ground-truth dataset of human preference judgments, measure judge-human agreement with Cohen's Kappa, and prompt-tune the judge to reduce systematic biases.
Calibrating Judge Models Against Humans is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Calibration Is Non-Negotiable
An uncalibrated LLM judge may systematically score outputs higher or lower than humans, prefer a specific writing style, or fail on edge cases your rubric did not anticipate. If you use such a judge to make deployment decisions, you are trusting a biased instrument. Calibration validates the judge against human ground truth and quantifies how much you can trust its scores before relying on them in production.
Building a Calibration Dataset
Create a calibration set of 100-500 example responses that have been scored by humans. Aim for diversity: include high-quality responses (score 5), medium quality (score 3), and clearly bad responses (score 1). Have 3-5 independent human raters score each example to measure inter-rater agreement and compute a consensus ground-truth score by averaging or taking the mode.
# Calibration dataset structure:
# [
# {
# 'question': 'What causes inflation?',
# 'response': '...',
# 'human_scores': [4, 4, 3, 5, 4], # 5 raters
# 'human_consensus': 4, # mean or mode
# 'rater_ids': ['r1', 'r2', 'r3', 'r4', 'r5']
# },
# ...
# ]
# Typical calibration set composition:
# 30% excellent responses (score 4-5)
# 40% adequate responses (score 2-4)
# 30% poor responses (score 1-2)
# Include adversarial / edge casesMeasuring Inter-Rater Agreement
Before trusting human scores as ground truth, verify that your human raters agree with each other. Cohen's Kappa measures agreement between two raters beyond chance: above 0.6 is acceptable, above 0.8 is excellent. For 5-point scales, compute weighted kappa (linear weights). Low inter-rater agreement means the task is ambiguously defined — improve rater guidelines before using the scores to calibrate a judge.
from sklearn.metrics import cohen_kappa_score
import numpy as np
def measure_inter_rater_agreement(rater1_scores: list, rater2_scores: list) -> dict:
kappa = cohen_kappa_score(rater1_scores, rater2_scores, weights='linear')
exact_agreement = sum(a == b for a, b in zip(rater1_scores, rater2_scores)) / len(rater1_scores)
adjacent_agreement = sum(abs(a - b) <= 1 for a, b in zip(rater1_scores, rater2_scores)) / len(rater1_scores)
return {
'weighted_kappa': round(kappa, 3),
'exact_agreement': round(exact_agreement, 3),
'adjacent_agreement': round(adjacent_agreement, 3),
'acceptable': kappa >= 0.6
}Running the Judge on Calibration Data
Run your LLM judge on every example in the calibration set using the same prompt you plan to use in production. Collect the judge's score for each example alongside the human consensus score. Keep these two lists aligned so you can compare them element-by-element. This pairwise comparison is what reveals systematic biases and score range differences.
async def run_judge_on_calibration(calibration_set: list) -> list:
pairs = []
for item in calibration_set:
judge_result = await async_judge(item['question'], item['response'])
pairs.append({
'question': item['question'],
'human': item['human_consensus'],
'judge': judge_result.correctness,
'judge_rationale': judge_result.rationale
})
return pairsComputing Judge-Human Correlation
Compute Pearson correlation between judge scores and human consensus scores. This measures linear agreement: a correlation of 1.0 means the judge perfectly tracks human rankings. Also compute mean absolute error (MAE) to understand the average score gap. A correlation above 0.7 and MAE below 0.8 points on a 5-point scale typically indicates a judge reliable enough for production use.
from scipy.stats import pearsonr, spearmanr
import numpy as np
def calibration_metrics(pairs: list) -> dict:
humans = [p['human'] for p in pairs]
judges = [p['judge'] for p in pairs]
pearson_r, p_val = pearsonr(humans, judges)
spearman_r, _ = spearmanr(humans, judges)
mae = np.mean([abs(h - j) for h, j in zip(humans, judges)])
return {
'pearson_r': round(pearson_r, 3),
'spearman_r': round(spearman_r, 3),
'p_value': round(p_val, 5),
'mae': round(mae, 3),
'reliable': pearson_r >= 0.7 and mae <= 0.8
}Identifying Systematic Biases
Beyond correlation, look for systematic biases: does the judge consistently over-score or under-score? Plot judge scores vs human scores and look for a regression line that does not pass through the origin. If the judge gives 4s where humans give 3s on average, it has an inflation bias. Correct for this by either adjusting the rubric or applying a linear calibration transform to the raw scores.
import numpy as np
from scipy import stats
def detect_score_bias(pairs: list) -> dict:
humans = np.array([p['human'] for p in pairs])
judges = np.array([p['judge'] for p in pairs])
slope, intercept, r, p, se = stats.linregress(judges, humans)
bias = np.mean(judges - humans) # positive = judge over-scores
return {
'mean_bias': round(bias, 3), # > 0 means judge inflates
'calibration_slope': round(slope, 3),
'calibration_intercept': round(intercept, 3),
# Corrected score = slope * judge_score + intercept
'correction_formula': f'human_score ≈ {slope:.2f} * judge + {intercept:.2f}'
}Applying Calibration Correction
Once you have the calibration regression (slope and intercept), apply it to transform raw judge scores into calibrated scores that better match human judgments. This linear correction is simple and effective. Clip the corrected score to the valid range (1-5) to prevent out-of-bounds values from the regression. Store both raw and corrected scores to allow retrospective comparison as calibration data grows.
def calibrate_score(raw_judge_score: float, slope: float, intercept: float,
min_score: int = 1, max_score: int = 5) -> float:
corrected = slope * raw_judge_score + intercept
return max(min_score, min(max_score, round(corrected, 1)))
# Example: if judge inflates by 0.5 points on average
# slope=0.85, intercept=0.3
# raw_score=4 -> corrected = 0.85*4 + 0.3 = 3.7Finding Systematic Error Patterns
Group calibration errors by question type, response length, and topic area to find patterned failures. The judge may be unreliable on technical coding questions but accurate on factual questions. It may over-score very long responses but under-score short concise answers. These patterns inform targeted rubric improvements or topic-specific judge prompts for the domains where calibration is weakest.
from collections import defaultdict
def error_by_category(pairs: list) -> dict:
by_cat = defaultdict(list)
for p in pairs:
error = abs(p['judge'] - p['human'])
by_cat[p.get('category', 'unknown')].append(error)
return {
cat: {
'mean_error': round(sum(errs)/len(errs), 2),
'max_error': max(errs),
'n': len(errs)
}
for cat, errs in by_cat.items()
}Re-calibrating as Data Grows
Calibration is not a one-time exercise. As you collect more human ratings and as model updates change judge behavior, re-run calibration quarterly. Track calibration metrics over time — if Pearson correlation drops below 0.7, investigate whether a model update changed the judge's scoring distribution. Automate the calibration pipeline so re-running it is a single command that produces an updated correction coefficient.
def run_calibration_pipeline(calibration_data: list) -> dict:
# 1. Measure human inter-rater agreement
ira = measure_inter_rater_agreement(
[d['rater_1'] for d in calibration_data],
[d['rater_2'] for d in calibration_data]
)
# 2. Run judge on all items
pairs = run_judge_on_calibration(calibration_data)
# 3. Compute correlation metrics
metrics = calibration_metrics(pairs)
# 4. Detect bias
bias = detect_score_bias(pairs)
return {'ira': ira, 'metrics': metrics, 'bias': bias}Documenting Your Judge Configuration
Document the judge model, prompt version, calibration dataset size and date, calibration metrics, and any correction coefficients in a judge configuration file checked into version control. This creates an audit trail so future team members understand why scores look the way they do, and so you can detect when metric changes are due to judge reconfiguration versus genuine quality changes.
# judge_config.yaml
# judge_model: gpt-4o-2025-01-01
# judge_prompt_version: v3.2
# calibration_date: 2026-05-15
# calibration_set_size: 312
# calibration_metrics:
# pearson_r: 0.81
# spearman_r: 0.79
# mae: 0.54
# bias_correction:
# slope: 0.88
# intercept: 0.45
# next_recalibration: 2026-08-15Dimension-Specific Judge Prompts
A single judge prompt scoring multiple dimensions at once often produces correlated scores — the judge gives high correctness because it gave high clarity, anchoring on its first impression. Dimension-specific prompts evaluate each criterion independently in a separate API call. This is more expensive but produces more reliable scores that are genuinely independent measurements. Use separate prompts for dimensions where you expect scores to vary independently.
async def multi_dimension_judge(question: str, answer: str) -> dict:
# Run each dimension as a separate judge call
correctness, helpfulness, clarity = await asyncio.gather(
judge_single_dimension(question, answer, 'correctness'),
judge_single_dimension(question, answer, 'helpfulness'),
judge_single_dimension(question, answer, 'clarity')
)
return {
'correctness': correctness,
'helpfulness': helpfulness,
'clarity': clarity,
'composite': 0.5 * correctness + 0.3 * helpfulness + 0.2 * clarity
}Quick Check
Test your understanding of calibrating LLM judge models against human ratings.
Lesson Recap
In this lesson you learned: calibration datasets with human consensus scores provide ground truth to measure judge reliability, Pearson correlation and MAE quantify how well the judge tracks human judgments, and linear bias correction adjusts for systematic over- or under-scoring. Next up we build a continuous evaluation pipeline integrated with CI/CD.
Frequently asked questions
Is the “Calibrating Judge Models Against Humans” lesson free?
Yes — the full text of “Calibrating Judge Models Against Humans” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Calibrating Judge Models Against Humans”?
Collect a ground-truth dataset of human preference judgments, measure judge-human agreement with Cohen's Kappa, and prompt-tune the judge to reduce systematic biases. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Calibrating Judge Models Against Humans” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- The LLM-as-Judge Pattern
- Pointwise and Pairwise Evaluation
- Calibrating Judge Models Against Humans
- Building a Continuous Evaluation Pipeline