CI/CD & Structured Extraction

This lesson fuses two exam scenarios that share a single backbone: determinism under automation. Scenario 5 (Claude Code for CI/CD) and Scenario 6 (Structured Data Extraction) both ask the same question — how do you get machine-parseable, trustworthy output with no human in the loop?

The answer is the same in both worlds:

• Headless / non-interactive execution so a pipeline can drive Claude.
• JSON output with a schema so downstream code can parse results.
• Validation loops that catch and repair structural errors before they propagate.

Master this and you cover a meaningful slice of D3 (Config & Workflows) and D4 (Structured Output) — together 40% of the exam.

A pipeline has no TTY and no human to answer prompts. Claude Code must run non-interactively. The flag for this is -p (also written --print): it runs a single prompt, prints the result, and exits.

Pair it with --output-format json so the pipeline gets a structured envelope instead of free prose. Without these two flags, the command hangs waiting for interactive input and the build stalls.

• -p / --print = non-interactive, required in pipelines.
• --output-format json = parseable result (optionally with a schema).

A subtle but heavily-tested rule: in CI, review code in an ISOLATED session, separate from the session that generated it.

Why? The generating session retains its own reasoning and is biased toward defending its work — same-session self-review is a top anti-pattern. A fresh, independent instance has no attachment to the output and challenges it honestly.

This mirrors the extraction-world rule that independent/fresh-instance review beats same-session self-review. The author won't fight its own conclusions; a clean reviewer will.

A CI reviewer that cries wolf gets ignored. The goal in Scenario 5 is to minimize false positives so developers trust the gate.

The lever is explicit criteria, not vague pleas. "Be more precise" changes nothing. "Flag a comment ONLY when it contradicts the code" gives the model a sharp, testable boundary.

On re-runs, don't re-litigate everything: include the prior results and report only new or still-unfixed issues. This keeps the signal clean across iterations.

A classic distractor: "use the Batch API to cut CI costs by 50%". Wrong for a pre-merge gate.

Message Batches are 50% cheaper with up to a 24h window, but they have NO latency SLA and do NOT support multi-turn tool calling. A blocking, time-sensitive check (like a PR gate) cannot wait an unbounded amount of time.

• Use Batch for non-blocking work: overnight reports, large audits, nightly extraction runs.
• Do NOT use Batch for pre-merge / blocking / interactive checks.

For batch jobs, custom_id correlates each request to its result, and you re-submit only the failures.

Now Scenario 6. To extract data reliably, do not ask for JSON in prose and hope. Use tool_use with a JSON Schema: this eliminates syntax errors and enforces required fields.

Set tool_choice to "any" to guarantee the model calls a tool (i.e. produces structured output) rather than free text. Use {"type":"tool","name":"X"} to force one specific extraction tool.

The single most-tested schema rule: mark a field required ONLY if it is always present in the source.

If you require a field that may be absent, the model has no legal way to satisfy the schema except to fabricate a value. You traded a missing field for a hallucinated one — far worse for a data pipeline.

For optional or open-ended values, prefer an enum with an "other" value plus a free-text detail field. This keeps the output structured while staying extensible for cases you didn't enumerate.

A schema constrains shape, not correctness. Wrap extraction in a validation loop (Pydantic-style validation is the canonical pattern).

When validation fails, use retry-with-feedback: send the model three things together — the original document, the wrong output it produced, and the exact validation error. This is precise enough to fix format, structural, and arithmetic mistakes.

Critical boundary: retry fixes FORMAT errors. It does NOT help when the information is simply ABSENT from the source — re-asking a document for data it never contained just invites a fabricated answer.

How do you catch a wrong arithmetic value the schema happily accepted? Build the check into the extraction itself.

The pattern: extract both the calculated_total (sum the line items) and the stated_total (the figure printed on the document). Then compare them in code.

If they diverge, you've detected a discrepancy deterministically — either a document error or an extraction error — and can flag, retry, or escalate. One number can lie silently; two numbers expose the lie.

Extraction without provenance is unauditable. For every extracted claim, keep a claim-to-source mapping: the source document name or URL, the supporting quote, and the publication date.

When two sources give conflicting figures, do not arbitrarily pick one — annotate the conflict. Often the dates resolve the apparent contradiction (one figure is simply newer).

And render by content type: tables for financials, prose for narrative, lists for technical findings. Provenance plus correct rendering is what makes the output trustworthy enough to automate on.

Before you let an extraction pipeline run unattended, validate it honestly. A headline like "97% accurate" can hide poor performance on one specific document type or one field.

The disciplined approach:

• Stratified random sampling across document types — not a convenience sample.
• Field-level confidence, calibrated on a labeled validation set, before automating.

Aggregate-only metrics are a known anti-pattern. A 97% average with 40% accuracy on tax forms is a production incident waiting to happen.

Apply the full picture to a realistic exam decision.

Key takeaways for CI/CD and structured extraction:

• Headless: -p / --print + --output-format json for non-interactive, parseable pipeline runs.
• Isolated review beats same-session self-review; explicit criteria minimize false positives; on re-runs report only new/unfixed issues.
• Batch API = 50% cheaper, no latency SLA, no multi-turn tools — for overnight jobs, NEVER blocking checks.
• Structured output: tool_use + JSON Schema; tool_choice:"any" guarantees a structured call.
• Required only if always present — requiring an absent field forces fabrication; use enum + 'other' + detail for extensibility.
• Retry-with-feedback (original doc + wrong output + exact error) fixes FORMAT errors, not ABSENT data.
• Self-correct with calculated vs stated totals; keep provenance (source, quote, date) and annotate conflicts.
• Validate with stratified sampling + field-level confidence — aggregate accuracy hides weak spots.