Iterative Refinement with Examples
2-4 input/output examples and test-driven iteration.
Iterative Refinement with Examples is a free Claude Architect lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Claude Architect learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Examples Beat Adjectives
When a prompt under-performs, architects reach for vague fixes like "be more precise" or "try harder". These rarely move the needle. The reliable lever is explicit criteria plus concrete examples.
Compare: "be more precise" versus "flag a comment only when it contradicts the code". The second tells the model exactly where the decision boundary sits. In this lesson you'll learn to drive a prompt to production quality using 2-4 targeted input/output examples and a tight test-and-iterate loop.
How Few-Shot Actually Works
Few-shot prompting attaches a small set of worked examples to your instructions. The key insight: the model generalizes from the examples — it does not merely copy them. Given 3 representative cases, it infers the underlying rule and applies it to unseen inputs.
Few-shot is strongest for four jobs:
- Consistency across many calls
- Edge cases that words alone describe poorly
- Output format the model should mirror
- Reducing hallucination by anchoring behavior
Aim for 2-4 examples per ambiguity — enough to define the pattern, few enough to keep context lean.
Anatomy of a Good Example
An effective few-shot example is a paired input and the exact output you want back. The output must match the real schema or shape you'll consume in production — same fields, same casing, same structure.
Below, each example shows an input comment plus the precise verdict. The model learns the boundary: flag only genuine contradictions, not style nits.
examples = [
{
"input": "# returns the user's age\n def get_name(u): return u.name",
"output": {"flag": True, "reason": "comment says age, code returns name"},
},
{
"input": "# sort ascending\n items.sort()",
"output": {"flag": False, "reason": "comment matches behavior"},
},
]
system = (
"Flag a comment ONLY when it contradicts the code. "
"Style or wording issues are not contradictions.\n\n"
"Examples:\n" + "\n".join(
f"INPUT: {e['input']}\nOUTPUT: {e['output']}" for e in examples
)
)Pair Criteria With Examples
Examples and explicit criteria are partners, not substitutes. Criteria state the rule; examples calibrate the gray zone the rule can't fully capture in prose.
A common architect mistake is dumping examples with no governing instruction. The model then over-fits to surface features of the samples. Always lead with a crisp criterion — "flag only when X contradicts Y" — then let 2-4 examples pin down where X and Y blur.
Rule of thumb: if you can't write the criterion in one sentence, your task is still under-specified, and more examples won't fix that.
Build a Test Set First
Iterative refinement is test-driven. Before tuning the prompt, assemble a small labeled set of representative inputs with known-correct outputs. This is your ground truth — every prompt change is judged against it, not against a gut feeling.
Keep the test set separate from your few-shot examples. If you tune on the same cases you teach with, you're memorizing, not generalizing.
test_cases = [
{"input": "# deletes the record\n def archive(r): r.archived = True",
"expected": {"flag": True}},
{"input": "# cache result for 60s\n cache.set(k, v, ttl=60)",
"expected": {"flag": False}},
{"input": "# returns count\n def total(rows): return sum(r.amt for r in rows)",
"expected": {"flag": True}},
]The Iteration Loop
The refinement loop is mechanical and repeatable:
- Run the prompt over every test case
- Compare output to the expected label
- Inspect the failures — what boundary did the model miss?
- Add or sharpen one example (or one criterion) that targets that failure
- Re-run the full set and check nothing regressed
Change one thing per iteration. Batch edits make it impossible to know which tweak helped or hurt.
Scoring the Test Set
Automate the comparison so iteration is fast. A tiny harness runs each case, scores it, and prints the misses. With structured output you can compare fields directly instead of parsing prose.
Note tool_choice below: forcing a tool call guarantees the model returns schema-valid JSON every time, so your scorer never trips over free-text.
def score(client, system, cases):
misses = []
for c in cases:
resp = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=256,
system=system,
tools=[verdict_tool],
tool_choice={"type": "any"}, # must call a tool -> structured output
messages=[{"role": "user", "content": c["input"]}],
)
out = resp.content[0].input
if out["flag"] != c["expected"]["flag"]:
misses.append((c["input"], out))
return missesTarget Failures With New Examples
When you read a miss, ask: which ambiguity caused it? Then add an example that lives exactly on that boundary — not a random new case. One sharp example per failure mode generalizes far better than ten generic ones.
Suppose the model wrongly flagged a comment that paraphrased the code. You'd add a paired example showing paraphrase = not a contradiction. The model updates its internal boundary and the whole class of similar misses clears.
Resist the urge to keep piling on examples. Past 4-ish per ambiguity you bloat context and risk lost-in-the-middle, where the model under-attends to the center of a long prompt.
# Add ONE example aimed at the observed miss:
examples.append({
"input": "# loop over each item\n for x in items: process(x)",
"output": {"flag": False,
"reason": "paraphrase of code, not a contradiction"},
})Format Errors? Retry With Feedback
Some failures are format or structural, not reasoning errors — a malformed field, a bad arithmetic total, a missing bracket. For these, retry-with-feedback is the fix: send back the original input, the wrong output, and the exact validation error.
Crucial limit: retry helps when the model can produce the right answer but slipped. It does not help when the required information is simply absent from the source — no amount of retrying invents data that isn't there.
def retry_with_feedback(client, system, original, bad_output, error):
return client.messages.create(
model="claude-sonnet-4-5",
max_tokens=512,
system=system,
messages=[
{"role": "user", "content": original},
{"role": "assistant", "content": str(bad_output)},
{"role": "user", "content":
f"That output failed validation: {error}. "
"Return corrected output that satisfies the schema."},
],
)Lock Quality With Structured Output
Once examples have shaped the behavior, lock the shape with a JSON Schema via tool use. This eliminates syntax errors and enforces required fields. One discipline matters most: mark a field required only if it is always present.
Never require a field that might be absent — the model will fabricate a value to satisfy the schema. For optional or open-ended data, use an enum with an "other" value plus a free-text detail field, which keeps the contract extensible without forcing hallucination.
verdict_tool = {
"name": "record_verdict",
"description": "Record whether a comment contradicts its code.",
"input_schema": {
"type": "object",
"properties": {
"flag": {"type": "boolean"},
"category": {"type": "string",
"enum": ["contradiction", "style", "other"]},
"detail": {"type": "string"},
},
"required": ["flag"], # only the always-present field
},
}Validate With a Fresh Reviewer
Before you trust the refined prompt, validate it with an independent, fresh instance — not the same session that produced the output. A same-session self-review is biased: the author retains its own reasoning and won't challenge itself.
And don't trust an aggregate score alone. A headline like 97% accuracy can hide a field or input type that's failing badly. Use stratified sampling and field-level confidence, calibrated on a labeled validation set, before you automate. Refinement isn't done when the average looks good — it's done when every slice clears the bar.
Quick Check: Fixing a Failing Prompt
A classifier prompt passes 92% of your labeled test set but consistently mislabels comments that paraphrase the code as contradictions. Which refinement move is most effective?
Recap: Refine With Examples, Prove With Tests
Key takeaways:
- Examples beat adjectives. Explicit criteria plus 2-4 targeted examples outperform vague instructions like 'be more precise'.
- The model generalizes from few-shot examples — best for consistency, edge cases, output format, and reducing hallucination.
- Be test-driven: build a labeled set, score it, add one boundary example per failure mode, re-run, change one thing at a time.
- Retry-with-feedback fixes format errors (send original + wrong output + exact error) — not missing-source data.
- Lock the shape with a JSON Schema; require only always-present fields; use enum + 'other' + detail for extensibility.
- Validate independently: fresh-instance review beats same-session self-review, and stratified field-level checks beat aggregate-only accuracy.
Frequently asked questions
Is the “Iterative Refinement with Examples” lesson free?
Yes — the full text of “Iterative Refinement with Examples” is free to read here on the web, and the Claude Architect course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Claude Architect course, upgrade to CoddyKit PRO.
What will I learn in “Iterative Refinement with Examples”?
2-4 input/output examples and test-driven iteration. You practise Claude Architect with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Claude Architect?
No prior experience is required. Claude Architect on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Iterative Refinement with Examples” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Claude Architect lesson?
Yes. Every Claude Architect lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Custom Commands vs Skills
- Skill Frontmatter
- Plan Mode vs Direct Execution
- Iterative Refinement with Examples