Критерии серьёзности с примерами

When you ask Claude to review code, the weakest instruction you can give is a vague one: be more precise or only flag important issues. The model has no shared definition of "important", so its bar drifts from file to file.

The exam principle is blunt: explicit criteria beat vague adjectives. A severity scale (Critical / High / Medium / Low) is only useful if each level has a written rule the model can apply consistently — and the most reliable way to pin down a rule is to anchor it with a concrete code example.

This lesson builds a severity rubric for a CI/CD review agent, one level at a time, each level tied to an example.

Here is the kind of prompt that looks fine but performs badly. It names severity levels but never defines them, so the model guesses — and guesses differently each run.

The result is exactly the anti-pattern the exam warns about: noisy reviews where a missing null-check and a misspelled comment both get tagged "High". Reviewers stop trusting the labels.

The repair is structural. For every severity level, give the model two things:

• A rule — a testable condition ("causes data loss, security breach, or a crash in production").
• An anchor example — a short snippet that unambiguously sits at that level.

This is few-shot prompting applied to a rubric: 2-4 targeted examples per ambiguity. The model generalizes from the anchors — it does not just echo them — so a handful of well-chosen examples calibrates the whole scale.

Critical is reserved for issues that cause data loss, a security breach, or a production crash. Anchor it with something undeniable — here, raw string interpolation into SQL.

Notice the anchor does double duty: it defines the ceiling of the scale, so the model knows nothing milder should reach this level.

High covers wrong behavior that won't crash the process but produces incorrect results — a logic error, a broken edge case, an off-by-one. The anchor makes the boundary with Critical concrete: no breach, no crash, but the output is wrong.

The low end of the scale is where vague prompts leak the most false positives, so anchor it just as carefully.

• Medium — maintainability or reliability risk that isn't yet a bug (a missing timeout, an unhandled-but-rare error path).
• Low — style and naming only; no behavioral impact.

Defining Low explicitly is what lets you later say "don't report Low in pre-merge gates" without the model arguing.

The anchored levels become one block in the system prompt. Keep this block stable and first — it's the same for every file you review, which makes it a perfect prompt-caching prefix. The per-file diff goes in the user turn, after the cached rubric.

A written rubric tells the model how to decide; structured output guarantees the shape of the answer. Bind severity to a JSON Schema enum so the field can never be a free-text adjective like "prettyBad".

Exam rule to remember: mark a field required only if it is always present. severity and line always exist for a real finding, so they are required; an optional suggested_fix is not.

Now combine the cached, anchored rubric with the enum-constrained schema in one request. The diff is the only volatile part, so it sits last in the user turn.

This pairing — explicit criteria for the decision, structured output for the format — is the exam's recommended pattern for reliable extraction and classification.

Once severity is a clean enum, the decision to block a merge is deterministic code, not a model judgment. The model classifies; your pipeline thresholds.

This mirrors the exam's hook principle: when a failure has real consequences (a broken merge), enforce it with deterministic code, not a probabilistic prompt. The rubric makes the model's labels trustworthy enough to gate on.

One subtle trap: telling the model at the finding stage to "only report Critical and High" suppresses recall — it silently drops issues it judged lower, and you lose coverage you might have wanted.

The robust pattern: have the model report every finding with its severity, then filter in a separate downstream step (your BLOCKING set, or an independent review pass). Coverage first, ranking second. Anchored criteria are what make that downstream ranking reliable.

Apply the lesson to a realistic design choice.

Key takeaways:

• Vague adjectives drift; written rules don't. Replace "important" with a testable condition per severity level.
• Anchor every level with a code example. 2-4 targeted few-shot anchors calibrate the scale — the model generalizes from them.
• Lock the label with an enum. Structured output makes severity always one of the valid values; require only fields that are always present.
• Cache the rubric, vary the diff. Keep the stable criteria first in the system prompt; put the per-file code last.
• Classify in the model, gate in code. Report all findings with severity, then filter/block deterministically downstream — never self-filter at the finding stage.