AI Prompt Engineering · Lesson

Zero, One, and Few-Shot

Choosing the number of examples.

Zero, One, and Few-Shot is a free AI Prompt Engineering lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Shot Spectrum

Shots denote the number of labeled demonstrations placed in the prompt before the live query. Zero-shot relies entirely on the model's pretrained priors; few-shot conditions the model on a small task-specific distribution at inference time without any weight updates.

This is in-context learning (ICL): the transformer treats the examples as part of the sequence and implicitly performs a kind of meta-learned regression over them. The choice of k (number of shots) is a hyperparameter you tune empirically, not a fixed best practice.

from dataclasses import dataclass

@dataclass
class ICLConfig:
    k: int           # number of demonstrations
    selection: str   # 'static' | 'dynamic'
    order: str       # 'random' | 'similarity' | 'curriculum'

# Zero-shot is simply k=0
cfg = ICLConfig(k=0, selection='static', order='random')

When Zero-Shot Wins

Prefer zero-shot when the task is well represented in pretraining (summarization, translation, common classification) and when examples would bias the output format. For instruction-tuned models, a crisp directive plus an output schema often beats examples that subtly anchor style.

Zero-shot also minimizes token cost and latency, and avoids majority-label bias where the model over-predicts whichever class dominates your demonstrations.

# Zero-shot with explicit schema beats vague few-shot
PROMPT = (
    'Classify sentiment as POSITIVE, NEGATIVE, or NEUTRAL.\n'
    'Respond with only the label.\n\n'
    'Text: ' + user_text + '\nLabel:'
)

One-Shot as Format Anchor

One-shot shines when the task is conceptually clear but the output format is unusual or strict. A single demonstration teaches the exact shape (JSON keys, delimiters, casing) far more reliably than a prose description.

Use one-shot when you want to constrain structure without spending tokens or risking the label-distribution skew that multiple examples introduce.

ONE_SHOT = (
    'Extract entities as JSON.\n\n'
    'Input: Apple released the iPhone in Cupertino.\n'
    'Output: {"org": ["Apple"], "product": ["iPhone"], "loc": ["Cupertino"]}\n\n'
    'Input: ' + query + '\nOutput:'
)

Few-Shot and the k Curve

Performance versus k is rarely monotonic. It typically rises, plateaus, then degrades as examples crowd the context, dilute attention, and push the live query further from the model's recency focus.

Empirically sweep k in {1, 2, 4, 8, 16} on a held-out set. The optimal k depends on task complexity, example length, and the model's effective context utilization, which is usually far below its advertised window.

def sweep_k(eval_set, candidates, ks=(1,2,4,8,16)):
    results = {}
    for k in ks:
        acc = evaluate(build_prompt(candidates[:k]), eval_set)
        results[k] = acc
    return max(results, key=results.get)

Why ICL Works: Implicit Inference

Research frames ICL as the model performing implicit Bayesian inference: demonstrations help locate the latent task concept the model already learned during pretraining. The examples act as evidence narrowing the posterior over tasks, not as new knowledge.

This explains a counterintuitive finding: even incorrect labels in demonstrations can preserve much of the accuracy, because the dominant signal is the format and label space, not the input-label mapping itself.

# Min, Lyu et al. (2022): label correctness matters less than
#   - the input distribution
#   - the label space (which classes exist)
#   - the format / structure
# Implication: invest in representative inputs + valid label set

Token Budget and Cost Tradeoffs

Every shot consumes context and money. With long demonstrations, four examples can dwarf the query. Compute a cost-per-accuracy-point metric: if k=8 buys 0.5% over k=4 at double the tokens, k=4 wins in production.

For high-throughput pipelines, prefer prompt caching of the static example block so repeated demonstrations are billed and processed once.

def cost_efficiency(acc_by_k, tokens_by_k, price_per_1k):
    return {
        k: acc_by_k[k] / (tokens_by_k[k] / 1000 * price_per_1k)
        for k in acc_by_k
    }
# Pick the k maximizing accuracy per dollar, not raw accuracy

Majority-Label and Position Bias

Few-shot prompts carry hidden biases. Majority-label bias makes the model favor the most frequent class in the demos. Recency bias over-weights the final example. Common-token bias favors tokens that appear often.

Calibration techniques such as contextual calibration estimate the model's prior on a content-free input (for example, the token N/A) and divide it out, dramatically stabilizing few-shot classifiers.

# Contextual calibration (Zhao et al. 2021)
p_cf = model_probs(prompt_with_input('N/A'))  # content-free prior
W = 1.0 / p_cf                                  # diagonal correction
def calibrated(probs):
    return normalize(W * probs)

Balancing the Demonstration Set

To counter majority-label bias, balance classes across demonstrations and vary their order. For k=4 binary classification, use 2 positive and 2 negative, shuffled, rather than 3:1.

For generation tasks, balance along the dimensions that matter (length, tone, difficulty) so the model does not collapse to a single mode it saw most often.

import random

def balanced_demos(pool, k, label_fn):
    by_label = {}
    for ex in pool:
        by_label.setdefault(label_fn(ex), []).append(ex)
    per = k // len(by_label)
    picks = [e for lst in by_label.values() for e in random.sample(lst, per)]
    random.shuffle(picks)
    return picks

Few-Shot vs Fine-Tuning

Few-shot is the right tool when the task changes often, data is scarce, or you cannot host a tuned model. Fine-tuning wins when you have thousands of examples, need lowest per-call latency, or want to bake in format so prompts stay short.

A common production path: prototype with few-shot, harvest the successful traces, then distill them into a fine-tune to drop the example tokens entirely.

# Decision heuristic
if num_labeled < 500 or task_volatility == 'high':
    strategy = 'few-shot ICL'
elif latency_budget_ms < 200 or prompt_token_cost_dominant:
    strategy = 'fine-tune + zero-shot'
else:
    strategy = 'few-shot now, distill later'

Reasoning Tasks Need More Than Shots

For multi-step reasoning, raw few-shot answer pairs can hurt: the model learns to jump straight to an answer it cannot justify. Pair few-shot with chain-of-thought demonstrations that show the reasoning trace, not just the final label.

The number of shots interacts with reasoning depth; often k=2 high-quality CoT exemplars beat k=8 answer-only ones on arithmetic and logic benchmarks.

COT_SHOT = (
    'Q: A shop had 23 apples, used 20, bought 6 more. How many now?\n'
    'A: Start 23, minus 20 leaves 3, plus 6 is 9. Answer: 9\n\n'
    'Q: ' + question + '\nA:'
)

An Evaluation Harness for k

Treat shot selection as an empirical search backed by a harness. Hold out a validation set, control for example order with multiple seeds, and report mean and variance, because few-shot accuracy can swing several points purely from ordering.

Log per-k token counts and latency so the final choice optimizes the full objective, not just accuracy.

def harness(pool, val, ks, seeds=5):
    report = {}
    for k in ks:
        accs = []
        for s in range(seeds):
            demos = balanced_demos(pool, k, label_fn)
            accs.append(evaluate(build_prompt(demos), val))
        report[k] = (mean(accs), stdev(accs))
    return report

Quick Check

Test your understanding of shot selection and ICL bias.

Recap

Key takeaways:

k is a tunable hyperparameter; sweep it and watch the rise-plateau-degrade curve.
Zero-shot suits well-known tasks; one-shot anchors strict formats; few-shot conditions on a task distribution.
ICL works by locating a pretrained task, so format and label space dominate over label correctness.
Counter majority-label, recency, and common-token biases with balancing, shuffling, and contextual calibration.
Optimize accuracy-per-dollar, pair reasoning tasks with CoT exemplars, and distill stable few-shot prompts into fine-tunes.

Frequently asked questions

Is the “Zero, One, and Few-Shot” lesson free?

Yes — the full text of “Zero, One, and Few-Shot” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Zero, One, and Few-Shot”?

Choosing the number of examples. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Zero, One, and Few-Shot” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering