집계 지표가 실패를 숨기는 경우

Your extraction pipeline reports 97% accuracy. The dashboard is green, the stakeholders are happy, and someone proposes turning off human review entirely.

Stop. A single aggregate number is one of the most dangerous artifacts in a production Claude system. That 97% is an average over a mixed population. Averages are excellent at smoothing away exactly the failures that hurt you most.

In this lesson you'll learn why aggregate-only accuracy is a documented anti-pattern, and what to measure instead before you automate human oversight away.

Imagine your pipeline processes three document types in equal volume. The blended score is 97%. Looks uniform, right?

But blended numbers are weighted by volume, not by risk. A small, high-stakes document type can be drowned out entirely. The aggregate tells you nothing about where the 3% of errors land — and in practice, errors are almost never spread evenly.

Here is the failure mode the exam wants you to recognize: aggregate accuracy can hide poor performance on a specific document type or field.

Contracts at 93.5% might be your highest-value, highest-liability documents. A handful of wrong clauses extracted from contracts can cost more than thousands of correct receipts ever saved. Yet the dashboard happily reports 97% and invites you to automate.

The number isn't wrong. It's just answering the wrong question. "How good are we on average?" is rarely the question that matters. "Where are we weakest, and how much does that weakness cost?" is.

It gets subtler. Even within a single document type, the failure may live in one field. A contract extractor can nail the parties, dates, and addresses — and quietly mangle the termination_clause or liability_cap 20% of the time.

Average those fields together and you still see a comfortable score. So stratify by both axes: by document type AND by field. The cell where a critical document type meets a critical field is where your real risk concentrates.

How do you surface these hidden cells? Not by sampling 100 random documents — random sampling reproduces your volume distribution, so rare-but-critical types get almost no coverage.

Use stratified random sampling: partition the population into strata (document type, source, field criticality), then sample randomly within each stratum. Now your low-volume contract type gets a statistically meaningful sample instead of three lucky draws.

This is the exam's recommended technique for catching what aggregates hide.

Stratified sampling tells you where you stand offline. To decide per document, in production whether to auto-accept or route to a human, you need field-level confidence calibrated on a labeled validation set.

"Calibrated" is the load-bearing word. A raw confidence-looking score means nothing until you've checked, against ground truth, that documents marked 0.9 are actually correct ~90% of the time. Calibrate first; only then can a threshold like "auto-accept above 0.97" mean what you think it means.

Calibrated confidence isn't the only signal. For numeric extraction you can have the model expose its own work so you can catch errors deterministically.

Extract both calculated_total (summed from line items) and stated_total (the printed total). When they disagree, you've found a discrepancy no aggregate metric would ever reveal — a verifiable, document-level red flag that routes straight to review.

When a low-confidence or discrepant document surfaces, be precise about the fix. Retry-with-feedback repairs format, structural, and arithmetic errors — send the original document, the wrong output, and the exact validation error back to the model.

But retry does not help when the information is simply absent from the source. If the contract never states a liability cap, no amount of re-prompting will conjure one — and a model that fabricates one is exactly the failure your metrics must catch. Absent data is an escalation case, not a retry case.

This connects to a structured-output rule that directly affects your metrics. Mark a schema field required only if it is always present. Never require a field that may be absent — the model will fabricate a value to satisfy the schema, and a fabricated value still counts as a confident answer.

That fabrication can sail straight past an aggregate accuracy check while corrupting your highest-stakes field. Make optional fields optional, and use an enum with an "other" value plus a free-text detail field for extensibility.

To investigate a hidden failure you must be able to trace any extracted claim back to its origin. Maintain claim-to-source mappings: source document name, the exact quote, location, and publication date.

When a stratified audit flags the contract stratum, provenance lets a reviewer jump straight to the quote that produced a bad liability_cap — instead of re-reading the whole document. Provenance turns "our metric looks off somewhere" into "this field, from this quote, on this page, is wrong."

Put it together into an automation decision. You earn the right to reduce human oversight on a stratum only when the evidence for that specific stratum supports it.

• Stratified audit shows the stratum meets target accuracy.
• Field-level confidence is calibrated on labeled data.
• Self-correction and provenance catch the residual errors.

And note what is not on this list: a model self-rated confidence (1-10) or sentiment score is a bad escalation trigger. Good triggers are threshold violations, policy gaps, no progress, and explicit human requests — not the model's own untrained self-assessment.

A scenario-style question on metrics and oversight.

Key takeaways:

• Aggregate accuracy hides per-type and per-field failures — it's volume-weighted, not risk-weighted.
• Stratify before you trust: stratified random sampling by document type and field surfaces weak cells that random sampling misses.
• Calibrate field-level confidence on a labeled validation set before using any threshold to auto-accept.
• Self-correction (extract calculated_total and stated_total) and provenance (claim-to-source quote, page, date) catch and explain residual errors.
• Don't require possibly-absent fields (fabrication) and don't escalate on self-rated confidence or sentiment — use threshold violations, policy gaps, no progress, and explicit human requests.

Earn automation per stratum, on evidence. The green dashboard is the start of the investigation, not the end.