Aggregate Metrics Hide Failures
97% overall can mask one failing document type.
Aggregate Metrics Hide Failures is a free Claude Architect lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Claude Architect learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Headline Number Lies
Your extraction pipeline reports 97% accuracy. The dashboard is green, the stakeholders are happy, and someone proposes turning off human review entirely.
Stop. A single aggregate number is one of the most dangerous artifacts in a production Claude system. That 97% is an average over a mixed population. Averages are excellent at smoothing away exactly the failures that hurt you most.
In this lesson you'll learn why aggregate-only accuracy is a documented anti-pattern, and what to measure instead before you automate human oversight away.
Anatomy of a Misleading Average
Imagine your pipeline processes three document types in equal volume. The blended score is 97%. Looks uniform, right?
But blended numbers are weighted by volume, not by risk. A small, high-stakes document type can be drowned out entirely. The aggregate tells you nothing about where the 3% of errors land — and in practice, errors are almost never spread evenly.
# Same 97% aggregate, two very different realities
docs = {
"invoices": {"n": 1000, "correct": 990}, # 99.0%
"receipts": {"n": 1000, "correct": 985}, # 98.5%
"contracts": {"n": 1000, "correct": 935}, # 93.5%
}
total = sum(d["n"] for d in docs.values())
hits = sum(d["correct"] for d in docs.values())
print(f"aggregate = {hits/total:.1%}") # 97.0% — hides contracts
for name, d in docs.items():
print(name, f"{d['correct']/d['n']:.1%}")One Failing Document Type
Here is the failure mode the exam wants you to recognize: aggregate accuracy can hide poor performance on a specific document type or field.
Contracts at 93.5% might be your highest-value, highest-liability documents. A handful of wrong clauses extracted from contracts can cost more than thousands of correct receipts ever saved. Yet the dashboard happily reports 97% and invites you to automate.
The number isn't wrong. It's just answering the wrong question. "How good are we on average?" is rarely the question that matters. "Where are we weakest, and how much does that weakness cost?" is.
Field-Level Failures Hide Even Deeper
It gets subtler. Even within a single document type, the failure may live in one field. A contract extractor can nail the parties, dates, and addresses — and quietly mangle the termination_clause or liability_cap 20% of the time.
Average those fields together and you still see a comfortable score. So stratify by both axes: by document type AND by field. The cell where a critical document type meets a critical field is where your real risk concentrates.
# Stratify a labeled validation set by (doc_type, field)
import collections
stats = collections.defaultdict(lambda: [0, 0]) # [correct, total]
for row in labeled_validation_set:
key = (row["doc_type"], row["field"])
stats[key][1] += 1
stats[key][0] += int(row["pred"] == row["gold"])
for (doc, field), (ok, n) in sorted(stats.items()):
acc = ok / n
flag = " <-- REVIEW" if acc < 0.95 else ""
print(f"{doc:10} {field:18} {acc:.1%} (n={n}){flag}")Stratified Random Sampling
How do you surface these hidden cells? Not by sampling 100 random documents — random sampling reproduces your volume distribution, so rare-but-critical types get almost no coverage.
Use stratified random sampling: partition the population into strata (document type, source, field criticality), then sample randomly within each stratum. Now your low-volume contract type gets a statistically meaningful sample instead of three lucky draws.
This is the exam's recommended technique for catching what aggregates hide.
from collections import defaultdict
import random
def stratified_sample(docs, key_fn, per_stratum=50):
strata = defaultdict(list)
for d in docs:
strata[key_fn(d)].append(d)
sample = []
for stratum, items in strata.items():
k = min(per_stratum, len(items))
sample += random.sample(items, k) # random WITHIN stratum
return sample
audit_set = stratified_sample(all_docs, key_fn=lambda d: d["doc_type"])Field-Level Confidence, Calibrated
Stratified sampling tells you where you stand offline. To decide per document, in production whether to auto-accept or route to a human, you need field-level confidence calibrated on a labeled validation set.
"Calibrated" is the load-bearing word. A raw confidence-looking score means nothing until you've checked, against ground truth, that documents marked 0.9 are actually correct ~90% of the time. Calibrate first; only then can a threshold like "auto-accept above 0.97" mean what you think it means.
# Calibrate, then gate per field
def route_extraction(field_name, value, confidence, thresholds):
# thresholds[field] derived from a LABELED validation set,
# tighter for high-stakes fields
if confidence >= thresholds[field_name]:
return "auto_accept"
return "human_review"
thresholds = {
"vendor_name": 0.92,
"liability_cap": 0.99, # critical -> stricter gate
"termination_clause": 0.99,
}Self-Correction Surfaces Discrepancies
Calibrated confidence isn't the only signal. For numeric extraction you can have the model expose its own work so you can catch errors deterministically.
Extract both calculated_total (summed from line items) and stated_total (the printed total). When they disagree, you've found a discrepancy no aggregate metric would ever reveal — a verifiable, document-level red flag that routes straight to review.
# Self-correction: extract both, compare deterministically
schema = {
"type": "object",
"properties": {
"line_items": {"type": "array", "items": {"type": "number"}},
"calculated_total": {"type": "number"}, # model sums line items
"stated_total": {"type": "number"} # printed on the doc
},
"required": ["line_items", "calculated_total", "stated_total"]
}
def needs_review(out):
return abs(out["calculated_total"] - out["stated_total"]) > 0.01Don't Retry What Calibration Reveals
When a low-confidence or discrepant document surfaces, be precise about the fix. Retry-with-feedback repairs format, structural, and arithmetic errors — send the original document, the wrong output, and the exact validation error back to the model.
But retry does not help when the information is simply absent from the source. If the contract never states a liability cap, no amount of re-prompting will conjure one — and a model that fabricates one is exactly the failure your metrics must catch. Absent data is an escalation case, not a retry case.
def handle_low_confidence(doc, output, error):
if error.kind in ("format", "arithmetic", "schema"):
# retry with original doc + wrong output + exact error
return retry_with_feedback(doc, output, error)
if error.kind == "absent":
# info not in source -> never retry; route to human
return escalate_to_human(doc, reason="field absent in source")Schemas Must Not Force Fabrication
This connects to a structured-output rule that directly affects your metrics. Mark a schema field required only if it is always present. Never require a field that may be absent — the model will fabricate a value to satisfy the schema, and a fabricated value still counts as a confident answer.
That fabrication can sail straight past an aggregate accuracy check while corrupting your highest-stakes field. Make optional fields optional, and use an enum with an "other" value plus a free-text detail field for extensibility.
{
"type": "object",
"properties": {
"vendor_name": {"type": "string"},
"liability_cap": {"type": ["number", "null"]},
"doc_category": {
"type": "string",
"enum": ["invoice", "receipt", "contract", "other"]
},
"category_detail": {"type": "string"}
},
"required": ["vendor_name", "doc_category"]
}Provenance Makes Failures Auditable
To investigate a hidden failure you must be able to trace any extracted claim back to its origin. Maintain claim-to-source mappings: source document name, the exact quote, location, and publication date.
When a stratified audit flags the contract stratum, provenance lets a reviewer jump straight to the quote that produced a bad liability_cap — instead of re-reading the whole document. Provenance turns "our metric looks off somewhere" into "this field, from this quote, on this page, is wrong."
extraction = {
"field": "liability_cap",
"value": 500000,
"provenance": {
"source_doc": "acme_msa_2026.pdf",
"quote": "liability shall not exceed five hundred thousand dollars",
"page": 7,
"published": "2026-01-15"
}
}Decide on Evidence, Not Vibes
Put it together into an automation decision. You earn the right to reduce human oversight on a stratum only when the evidence for that specific stratum supports it.
- Stratified audit shows the stratum meets target accuracy.
- Field-level confidence is calibrated on labeled data.
- Self-correction and provenance catch the residual errors.
And note what is not on this list: a model self-rated confidence (1-10) or sentiment score is a bad escalation trigger. Good triggers are threshold violations, policy gaps, no progress, and explicit human requests — not the model's own untrained self-assessment.
Quick Check: Reading the 97%
A scenario-style question on metrics and oversight.
Recap: Make Hidden Failures Visible
Key takeaways:
- Aggregate accuracy hides per-type and per-field failures — it's volume-weighted, not risk-weighted.
- Stratify before you trust: stratified random sampling by document type and field surfaces weak cells that random sampling misses.
- Calibrate field-level confidence on a labeled validation set before using any threshold to auto-accept.
- Self-correction (extract calculated_total and stated_total) and provenance (claim-to-source quote, page, date) catch and explain residual errors.
- Don't require possibly-absent fields (fabrication) and don't escalate on self-rated confidence or sentiment — use threshold violations, policy gaps, no progress, and explicit human requests.
Earn automation per stratum, on evidence. The green dashboard is the start of the investigation, not the end.
Frequently asked questions
Is the “Aggregate Metrics Hide Failures” lesson free?
Yes — the full text of “Aggregate Metrics Hide Failures” is free to read here on the web, and the Claude Architect course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Claude Architect course, upgrade to CoddyKit PRO.
What will I learn in “Aggregate Metrics Hide Failures”?
97% overall can mask one failing document type. You practise Claude Architect with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Claude Architect?
No prior experience is required. Claude Architect on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Aggregate Metrics Hide Failures” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Claude Architect lesson?
Yes. Every Claude Architect lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Claim to Source Mappings
- Conflicting Data & Dates
- Aggregate Metrics Hide Failures
- Stratified Sampling & Calibration