AI Prompt Engineering · Lesson

Self-Critique Validation

Model-checked outputs.

Self-Critique Validation is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Model as Its Own Critic

Self-critique uses an LLM to evaluate an LLM's output against a rubric or policy. It catches nuanced failures that deterministic validators cannot encode: factual inconsistency, tone, helpfulness, subtle policy violations.

It is the model-based complement to schema and rule validators.

Separate the Critic from the Author

Run critique as a distinct call with its own prompt, not as a tail instruction in the generation. A clean-context critic, asked only to judge, is far more reliable than asking the author to grade itself mid-generation, where it is biased toward its own answer.

draft = author_model(task_prompt)
verdict = critic_model(
  'You are a strict reviewer. Judge ONLY the answer below against the rubric.\n'
  'Rubric: ' + rubric + '\nAnswer: ' + draft
)

Structured Critique Output

Make the critic emit structured verdicts so the pipeline can act programmatically. A free-text critique is not machine-actionable.

CRITIC_SCHEMA = {
  'type': 'object',
  'properties': {
    'pass': {'type': 'boolean'},
    'violations': {'type': 'array', 'items': {'type': 'string'}},
    'severity': {'type': 'string', 'enum': ['none','minor','major','critical']},
    'fix_hint': {'type': 'string'}
  },
  'required': ['pass','violations','severity','fix_hint'],
  'additionalProperties': False
}

Critique-then-Revise Loop

Pair the critic with a reviser. The critic finds problems; the author revises using the critique; repeat until pass or budget exhausted. This is the model-based analogue of the repair loop.

draft = author_model(task)
for _ in range(2):
    c = critic_model(draft)
    if c['pass']:
        break
    draft = author_model(task + '\nRevise to fix: ' + c['fix_hint'])
final = draft

Rubrics Make Critique Reliable

A vague instruction ('is this good?') yields noisy verdicts. A concrete rubric with explicit, checkable criteria yields consistent ones. Decompose into yes/no questions the critic answers individually.

RUBRIC = [
  'Does the answer directly address the user question?',
  'Are all factual claims supported by the provided context?',
  'Is any disallowed content present?',
  'Is the response within the requested length?'
]

Grounded Critique for Factuality

For hallucination detection, give the critic the source context and ask it to flag any claim not entailed by it. This turns self-critique into an entailment check, far stronger than asking 'is this true?' without evidence.

verdict = critic_model(
  'For each claim in the ANSWER, state whether the CONTEXT entails it. '
  'Flag any unsupported claim.\nCONTEXT:\n' + ctx + '\nANSWER:\n' + draft
)

Critic Failure Modes

The critic is itself an LLM and can fail:

Sycophancy — rubber-stamping the author's answer.
Over-criticism — flagging correct output.
Shared blind spots — same model misses the same errors.

Mitigate with a different model family as critic, a strict reviewer persona, and calibrated thresholds.

Use a Cheaper or Different Critic

The critic need not be the most expensive model. Often a smaller model with a tight rubric is a cost-effective gate, and using a different model family reduces correlated blind spots. Reserve the strongest model for authoring.

When to Trust vs Verify

Self-critique reduces error but is not a proof. For low-stakes content, a single critique pass is enough. For high-stakes outputs, combine self-critique with deterministic validators and human review; never let a model be the sole arbiter of safety-critical decisions.

Cost, Latency, and Caching

Self-critique roughly doubles calls per request. Control it: gate critique behind deterministic checks (only critique what passed schema/rules), cache critiques for identical drafts, and cap revision rounds. Run critique in parallel with non-blocking work when possible.

if deterministic_ok(draft):
    verdict = critic_model(draft)   # only spend critique on viable drafts

Calibrate Against Human Labels

Validate the critic before trusting it. Build a set of human-labeled outputs, run the critic, and measure agreement (precision/recall against human verdicts). Tune the rubric and persona until the critic's judgments correlate with humans; re-check after model upgrades.

agreement = mean(critic(d)['pass'] == human_label[d] for d in eval_set)
assert agreement > 0.9

Quick Check

You want the critic to reliably catch hallucinations in a RAG answer. What most improves reliability?

Recap

Self-critique validation:

A separate critic with a clean context judges the author's output.
Emit structured verdicts; drive a critique-then-revise loop.
Concrete rubrics and grounded entailment checks boost reliability.
Beware sycophancy and shared blind spots; use a different critic model.
Gate by cost, combine with deterministic checks, calibrate to humans.

You have completed guardrails. Next course: red-teaming and adversarial evaluation.

Frequently asked questions

Is the “Self-Critique Validation” lesson free?

Yes — the full text of “Self-Critique Validation” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Self-Critique Validation”?

Model-checked outputs. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Self-Critique Validation” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering