LLM-as-a-Judge Pitfalls
Judges are biased toward verbose answers, struggle with their own outputs, and need careful calibration.
LLM-as-a-Judge Pitfalls is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why LLM Judges?
For open-ended outputs (essays, code reviews, conversational answers), there is no single correct answer. Hiring humans to grade thousands of outputs is expensive.
LLM-as-a-Judge — using a strong model to score outputs — fills the gap.
Basic LLM Judge
judge_prompt = '''
You are an evaluator. On a scale of 1-5, rate the following answer
for accuracy and helpfulness. Return JSON: {score: int, reason: str}.
Question: {question}
Answer: {answer}
'''Pitfall 1: Position Bias
Judges prefer the FIRST answer in a pairwise comparison. Always randomize order and run twice:
def fair_compare(a, b):
score_ab = compare(a, b)
score_ba = compare(b, a) # swapped
return (score_ab + score_ba) / 2Pitfall 2: Length Bias
Judges prefer LONGER answers, even when shorter ones are better. Calibrate by normalising or instructing the judge:
judge_prompt = '... Penalize answers that are verbose without adding value ...'Pitfall 3: Self-Preference
Models prefer their own outputs. If GPT-4 judges its own work, it scores itself higher than it should. Use a different model family for judging:
- GPT-4 outputs -> judged by Claude
- Claude outputs -> judged by GPT-4
Pitfall 4: Sycophancy
Judges agree with strong opinions in the answer. "I am 100% sure..." gets higher scores than "I think...". Strip confidence words before judging.
Pitfall 5: Score Inflation
On 1-5 scales, judges cluster around 4. Use binary scoring (correct/incorrect) for sharper signal:
judge_prompt = '... Return PASS or FAIL only ...'Pitfall 6: Format Bias
Bullet-list answers score higher than prose, regardless of correctness. Be aware; sometimes mandate format to remove the variable.
Calibrate Against Humans
Measure how well the judge correlates with human ratings on a sample:
from scipy.stats import pearsonr
human_scores = [...]
llm_scores = [...]
rho, _ = pearsonr(human_scores, llm_scores)
print(f'Correlation: {rho}')
# Below ~0.6 — judge unreliable for this taskUse Pairwise When Possible
"Is A better than B?" is more reliable than "Score A from 1-5". When choosing between two prompts, pairwise > absolute.
Chain-of-Thought Judging
Make the judge reason first:
judge_prompt = '''
First list the factual claims in the answer.
Then check each against the reference.
Then produce a score.
Question: {q}
Reference: {r}
Answer: {a}
'''Cost
Judging with GPT-4 doubles eval cost. For routine checks, use cheaper judges (gpt-4o-mini or claude-haiku) and reserve big judges for releases.
Combine With Rules
Don't rely only on the judge. Combine:
- Rules ("contains '30 days'")
- Heuristics (length range)
- LLM judge for the open-ended part
Calibrating an LLM Judge
How do you check if your LLM judge is reliable?
Recap
LLM judges are useful but biased. Randomize positions, normalize length, use different model families, calibrate against humans, combine with hard rules.
Frequently asked questions
Is the “LLM-as-a-Judge Pitfalls” lesson free?
Yes — the full text of “LLM-as-a-Judge Pitfalls” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “LLM-as-a-Judge Pitfalls”?
Judges are biased toward verbose answers, struggle with their own outputs, and need careful calibration. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “LLM-as-a-Judge Pitfalls” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.