Stratified Sampling & Calibration

You ship a Claude extraction pipeline and report 97% accuracy. Leadership approves full automation. Three weeks later, every refund invoice with a foreign-currency line is wrong.

The headline number was real — but it was an average. Aggregate accuracy can hide catastrophic failure on a specific document type or a specific field, because the common cases drown out the rare ones.

This lesson is about the discipline that keeps you honest before you automate: stratified sampling to measure where you actually fail, and calibration on labeled validation sets so your confidence scores mean something.

Memorize the exam-level principle from the oversight domain:

Aggregate accuracy can hide poor performance on a specific document type or field. Use stratified random sampling plus field-level confidence calibrated on labeled validation sets before automating.

Two moves, in order:

• Stratify, then sample — partition the population into segments, sample within each, so rare-but-critical slices are actually measured.
• Calibrate confidence — make the model's per-field confidence correspond to real-world correctness, proven against ground-truth labels.

Skip either step and you are guessing, not governing.

Plain random sampling oversamples whatever is common. If 95% of your invoices are simple single-currency receipts, a random sample of 200 will be ~190 easy cases and maybe a handful of the hard foreign-currency ones — too few to detect a 40% error rate on that slice.

Stratified sampling fixes this: define segments that matter (document type, language, vendor, field presence), then draw a sample from each segment. Now the rare slice is measured with enough statistical power to fail loudly if it's broken.

You measure the population you're worried about, not the population that happens to be biggest.

Good strata isolate the dimensions where behavior plausibly diverges. For a structured-extraction pipeline (Scenario 6), useful segments include:

• Document type — invoice vs. receipt vs. statement.
• Field — totals, dates, currency, line items. Field-level matters because 97% overall can mask 60% on the currency field.
• Edge conditions — multi-page docs, scanned/low-quality, multilingual, or records where an optional field is absent.

That last one ties to a schema rule: never mark a possibly-absent field as required, or the model fabricates it. Stratify by presence/absence so you catch fabrication.

Concretely, group your validation pool by segment and pull a fixed quota from each — enough per stratum to trust the per-stratum metric, not proportional to the population.

To calibrate, the model has to emit a confidence per field, not one blanket score. Force structured output so confidence is a typed field you can audit — not prose you parse.

Use tool_choice to guarantee the model returns the schema. "any" forces it to call some tool; {"type":"tool","name":"X"} forces a specific one. A JSON Schema eliminates syntax errors and enforces required fields.

A model is calibrated when its stated confidence matches observed accuracy: of all fields it labels 0.90 confident, about 90% should be correct against ground truth.

Raw model confidence is usually over-confident — it says 0.95 but is right 70% of the time. You can't trust an auto-accept threshold built on uncalibrated scores; you'd auto-approve wrong data.

Calibration requires labeled validation sets: human-verified ground truth for your stratified sample. There is no shortcut — the model's self-rated confidence alone is exactly the kind of signal the exam tells you NOT to trust for automated decisions.

Bucket predictions by stated confidence, then compare each bucket's stated confidence to its actual accuracy on the labeled set. The gap is your calibration error — and you compute it per stratum so a well-calibrated 'easy' slice can't mask a broken 'hard' one.

Once calibrated, derive a confidence threshold for each stratum that meets your accuracy bar — say, auto-accept only where calibrated accuracy ≥ 99%. The threshold will differ by segment: easy single-currency invoices might auto-accept at 0.85, while foreign-currency lines need 0.98 — or stay human-reviewed entirely.

This is the bridge from measurement to oversight: low-confidence or low-accuracy strata route to a human; high-confidence calibrated strata automate. A single global threshold would either over-trust the hard slice or needlessly gate the easy one.

Confidence improves when the model can cross-check itself. For numeric extraction, have it emit both a calculated_total (sum of line items) and the stated_total printed on the document, then flag any discrepancy. A mismatch is a hard, deterministic signal — far more reliable than a self-rated confidence number.

Pair this with provenance: keep each claim's source (doc name, quote, page) so a flagged field can be traced and verified, not just re-guessed.

Stratified calibration audits are large, non-blocking jobs — exactly what the Message Batches API is for: 50% cheaper, up to a 24-hour window, no latency SLA. Run your overnight calibration sweep as a batch; correlate results with custom_id and re-submit only the failures.

The boundary matters for the exam: never use the Batch API for blocking or time-sensitive checks (it has no latency SLA and does not support multi-turn tool calling). A pre-merge gate or a live extraction stays on the synchronous API; the periodic audit goes to batch.

Apply the rule to a real go/no-go decision.

Key takeaways for the exam and for production:

• Aggregate accuracy hides per-type and per-field failure — never automate on a headline average alone.
• Stratified random sampling partitions by the dimensions that matter (doc type, field, edge conditions) and samples each, giving rare slices real statistical power.
• Calibration aligns stated confidence with observed accuracy, proven on a labeled validation set; raw model confidence is over-confident and self-rated scores are not a trustworthy automation signal.
• Set auto-accept thresholds per segment; automate calibrated high-accuracy strata, route the rest to humans.
• Strengthen the signal with self-correction (calculated vs. stated total) and provenance; run the heavy calibration audit on the Batch API — never on a blocking, time-sensitive path.