การปรับปรุงซ้ำด้วยตัวอย่าง

When a prompt under-performs, architects reach for vague fixes like "be more precise" or "try harder". These rarely move the needle. The reliable lever is explicit criteria plus concrete examples.

Compare: "be more precise" versus "flag a comment only when it contradicts the code". The second tells the model exactly where the decision boundary sits. In this lesson you'll learn to drive a prompt to production quality using 2-4 targeted input/output examples and a tight test-and-iterate loop.

Few-shot prompting attaches a small set of worked examples to your instructions. The key insight: the model generalizes from the examples — it does not merely copy them. Given 3 representative cases, it infers the underlying rule and applies it to unseen inputs.

Few-shot is strongest for four jobs:

• Consistency across many calls
• Edge cases that words alone describe poorly
• Output format the model should mirror
• Reducing hallucination by anchoring behavior

Aim for 2-4 examples per ambiguity — enough to define the pattern, few enough to keep context lean.

An effective few-shot example is a paired input and the exact output you want back. The output must match the real schema or shape you'll consume in production — same fields, same casing, same structure.

Below, each example shows an input comment plus the precise verdict. The model learns the boundary: flag only genuine contradictions, not style nits.

Examples and explicit criteria are partners, not substitutes. Criteria state the rule; examples calibrate the gray zone the rule can't fully capture in prose.

A common architect mistake is dumping examples with no governing instruction. The model then over-fits to surface features of the samples. Always lead with a crisp criterion — "flag only when X contradicts Y" — then let 2-4 examples pin down where X and Y blur.

Rule of thumb: if you can't write the criterion in one sentence, your task is still under-specified, and more examples won't fix that.

Iterative refinement is test-driven. Before tuning the prompt, assemble a small labeled set of representative inputs with known-correct outputs. This is your ground truth — every prompt change is judged against it, not against a gut feeling.

Keep the test set separate from your few-shot examples. If you tune on the same cases you teach with, you're memorizing, not generalizing.

The refinement loop is mechanical and repeatable:

• Run the prompt over every test case
• Compare output to the expected label
• Inspect the failures — what boundary did the model miss?
• Add or sharpen one example (or one criterion) that targets that failure
• Re-run the full set and check nothing regressed

Change one thing per iteration. Batch edits make it impossible to know which tweak helped or hurt.

Automate the comparison so iteration is fast. A tiny harness runs each case, scores it, and prints the misses. With structured output you can compare fields directly instead of parsing prose.

Note tool_choice below: forcing a tool call guarantees the model returns schema-valid JSON every time, so your scorer never trips over free-text.

When you read a miss, ask: which ambiguity caused it? Then add an example that lives exactly on that boundary — not a random new case. One sharp example per failure mode generalizes far better than ten generic ones.

Suppose the model wrongly flagged a comment that paraphrased the code. You'd add a paired example showing paraphrase = not a contradiction. The model updates its internal boundary and the whole class of similar misses clears.

Resist the urge to keep piling on examples. Past 4-ish per ambiguity you bloat context and risk lost-in-the-middle, where the model under-attends to the center of a long prompt.

Some failures are format or structural, not reasoning errors — a malformed field, a bad arithmetic total, a missing bracket. For these, retry-with-feedback is the fix: send back the original input, the wrong output, and the exact validation error.

Crucial limit: retry helps when the model can produce the right answer but slipped. It does not help when the required information is simply absent from the source — no amount of retrying invents data that isn't there.

Once examples have shaped the behavior, lock the shape with a JSON Schema via tool use. This eliminates syntax errors and enforces required fields. One discipline matters most: mark a field required only if it is always present.

Never require a field that might be absent — the model will fabricate a value to satisfy the schema. For optional or open-ended data, use an enum with an "other" value plus a free-text detail field, which keeps the contract extensible without forcing hallucination.

Before you trust the refined prompt, validate it with an independent, fresh instance — not the same session that produced the output. A same-session self-review is biased: the author retains its own reasoning and won't challenge itself.

And don't trust an aggregate score alone. A headline like 97% accuracy can hide a field or input type that's failing badly. Use stratified sampling and field-level confidence, calibrated on a labeled validation set, before you automate. Refinement isn't done when the average looks good — it's done when every slice clears the bar.

A classifier prompt passes 92% of your labeled test set but consistently mislabels comments that paraphrase the code as contradictions. Which refinement move is most effective?

Key takeaways:

• Examples beat adjectives. Explicit criteria plus 2-4 targeted examples outperform vague instructions like 'be more precise'.
• The model generalizes from few-shot examples — best for consistency, edge cases, output format, and reducing hallucination.
• Be test-driven: build a labeled set, score it, add one boundary example per failure mode, re-run, change one thing at a time.
• Retry-with-feedback fixes format errors (send original + wrong output + exact error) — not missing-source data.
• Lock the shape with a JSON Schema; require only always-present fields; use enum + 'other' + detail for extensibility.
• Validate independently: fresh-instance review beats same-session self-review, and stratified field-level checks beat aggregate-only accuracy.