Reducing False Positives

You wired Claude Code into your CI pipeline to review every pull request. It works — but developers start ignoring it. Why? Too much noise. The review keeps flagging things that are not real problems: style nitpicks, harmless TODO comments, defensive null checks.

This is the false-positive problem. A reviewer that cries wolf gets muted. In Scenario 5 (Claude Code for CI/CD), one of the core architect skills is minimizing false positives so the signal that remains is trustworthy.

In this lesson you'll learn a fast, surgical tactic: temporarily disable high-noise categories so the review stays useful while you tune the real rules.

A CI reviewer has exactly one job: surface issues a human should act on. The moment it produces more false alarms than real findings, two things happen:

• Developers stop reading the comments.
• Real bugs hide inside the noise (lost-in-the-middle: attention drops on the long middle of a list).

Aggregate "we found 40 issues" sounds productive, but if 35 are noise the review has negative value — it costs attention and returns little. Reducing false positives is not cosmetic; it protects the credibility of the whole pipeline.

Before disabling anything, find which categories generate the noise. Run the review in non-interactive mode with machine-readable output so you can count findings by type instead of eyeballing prose.

Use -p (also --print) for non-interactive CI runs and --output-format json for parseable results.

Once you spot a category producing mostly false positives — say style or doc_formatting — the fastest fix is to temporarily disable it. Not delete it forever; mute it now, re-enable once you've written sharper criteria.

The cleanest place to do this is the instruction the reviewer reads. Be explicit about what to skip, because explicit criteria beat vague pleas like "be less noisy."

A flag on one command only mutes one run. To make the rule consistent across every CI review, put it in the project-level CLAUDE.md (./CLAUDE.md or .claude/CLAUDE.md) — shared via version control so every teammate and every pipeline run inherits it.

Avoid user-level ~/.claude/CLAUDE.md for this: it is personal and NOT shared via VCS, so new teammates and the CI runner would miss the rule.

Sometimes a category is noisy only in part of the repo. Generated files or test fixtures, for example, will trip a strict reviewer constantly. Instead of bloating the monolithic CLAUDE.md, use a .claude/rules/ file with YAML frontmatter paths — it loads only when the matching files are in play, saving context and tokens.

Muting noisy categories buys you breathing room, but the durable fix is sharper criteria for the categories you keep. Explicit criteria beat vague instructions every time.

• Vague: "Flag bad comments." → fires on everything.
• Explicit: "Flag a comment ONLY when it contradicts the code it describes." → fires on real defects.

Re-enabling a category with a precise rule is far better than leaving it muted forever.

When a category keeps misfiring on borderline cases, don't just describe the boundary — show it. Two to four targeted few-shot examples per ambiguity teach the model the line between signal and noise. The model generalizes from them; it does not merely copy them.

Few-shot is especially strong for consistency, edge cases, output format, and reducing hallucinated findings — exactly the levers that drive false positives down.

One subtle false-positive source: bias. If the same session that generated code also reviews it, the author retains its reasoning and won't challenge itself — it rationalizes its own choices. The same applies to a reviewer primed by long generation context.

Run the review in a fresh, isolated session. An independent instance evaluates the diff on its own merits and produces cleaner, less self-justifying findings.

A noisy pattern on iterative PRs: the reviewer re-reports the same findings every push, drowning the genuinely new ones. When you re-run a review, feed it the prior results and ask it to report only new or still-unfixed issues.

This keeps each run's output tight and stops developers from scrolling past repeats — another quiet driver of false-positive fatigue.

Disabling a noisy category is a tuning decision about review signal — it is NOT how you enforce critical rules. Prompt-level instructions are roughly 90% probabilistic; they are perfect for shaping what a review reports, but wrong for guarantees.

When a rule has financial, legal, or safety consequences (block a refund over $500, reject a secret committed to the repo), enforce it with a deterministic hook, not a prompt. Mute noise with prompts and config; enforce hard rules with hooks. Keep the two jobs separate.

Apply what you've learned to a realistic CI situation.

Key takeaways for keeping a CI reviewer trustworthy:

• Noise destroys trust — a reviewer that cries wolf gets muted, and real bugs hide in the list.
• Measure first — run with -p and --output-format json, then tally findings by category to locate the noise.
• Temporarily disable high-noise categories; mute now, re-enable with sharper rules later.
• Encode it in project CLAUDE.md (shared via VCS), and use .claude/rules/ with paths to scope mutes to generated or fixture files.
• Sharpen the survivors with explicit criteria and 2-4 few-shot examples per ambiguity.
• Review in an isolated session and report only new/unfixed issues on re-runs.
• Mute with prompts/config; enforce critical rules with deterministic hooks. Never confuse the two.