AI Prompt Engineering · Lesson

Example Ordering and Recency

How example order affects output.

Example Ordering and Recency is a free AI Prompt Engineering lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Order Is a Hidden Hyperparameter

The order of few-shot examples can swing accuracy by several percentage points, sometimes turning a near-random prompt into a strong one and vice versa. This volatility exists even when the example set is fixed.

Treat ordering as a first-class hyperparameter to search, not an afterthought. Reporting few-shot results without controlling for order is scientifically unsound.

import itertools

def order_search(demos, eval_set, build, max_perms=24):
    perms = list(itertools.permutations(demos))[:max_perms]
    scored = [(p, evaluate(build(p), eval_set)) for p in perms]
    return max(scored, key=lambda x: x[1])

Recency Bias Explained

Decoder-only transformers exhibit a recency bias: tokens nearer the generation point exert disproportionate influence. The last demonstration before the query is the most likely to shape format, label, and tone.

This is partly an artifact of attention patterns and positional encoding. The practical upshot: whatever you place last is what the model most readily imitates.

# Position weight intuition (illustrative, not exact)
# influence(example_i) roughly increases with proximity to the query
# => the FINAL demo disproportionately anchors the next generation
weights = [0.1, 0.15, 0.25, 0.5]  # earlier ... latest

Lost in the Middle

Long-context studies reveal a U-shaped attention profile: models attend well to the beginning and end of the context but under-attend the middle. Critical demonstrations buried mid-prompt are effectively discounted.

Place your most important or trickiest examples at the start or, given recency bias, near the end, never stranded in the middle of a long demo block.

def place_key_demos(key, filler):
    # U-shape aware: key items at the edges, filler in the middle
    head = key[: len(key)//2]
    tail = key[len(key)//2 :]
    return head + filler + tail

Majority Label and Last Position

Recency and majority-label biases compound. If your final demonstration is class A, the model leans toward predicting A. Combined with an A-heavy demo set, the prior becomes strongly skewed.

For balanced classification, alternate labels and ensure the final example does not always belong to the same class across your prompt variants.

def alternate_labels(by_label, k):
    labels = list(by_label)
    out = []
    i = 0
    while len(out) < k:
        lbl = labels[i % len(labels)]
        if by_label[lbl]:
            out.append(by_label[lbl].pop())
        i += 1
    return out  # final element varies by construction

Similarity-Ordered (Ascending) Demos

A robust heuristic in retrieval-augmented few-shot: order retrieved examples by ascending similarity to the query, so the most relevant demonstration sits closest to the query and benefits from recency.

This pairs dynamic selection with recency exploitation: relevant context is both present and positioned where the model attends most strongly.

def order_by_similarity(query_emb, retrieved):
    scored = [(d, cos(query_emb, d.emb)) for d in retrieved]
    scored.sort(key=lambda x: x[1])     # ascending
    return [d for d, _ in scored]        # most similar last (near query)

Curriculum Ordering

For reasoning or generation, a curriculum order (easy to hard) can help the model build up the target behavior, ending on a complex exemplar that demonstrates the full reasoning depth right before the query.

This is task-dependent; validate it against random and similarity orderings rather than assuming it helps.

def curriculum(demos, difficulty_fn):
    return sorted(demos, key=difficulty_fn)  # easiest first, hardest last

Measuring Order Sensitivity

Quantify how brittle your prompt is by sampling many orderings and reporting the variance of accuracy. High variance signals an unstable prompt that may regress unpredictably in production.

Use this metric to compare prompt designs: a prompt that is order-robust is safer than a marginally higher-scoring but volatile one.

import random

def order_sensitivity(demos, eval_set, build, trials=20):
    accs = []
    for _ in range(trials):
        perm = random.sample(demos, len(demos))
        accs.append(evaluate(build(perm), eval_set))
    return mean(accs), stdev(accs)  # report both; low stdev = robust

Probing-Free Order Selection

When you lack labels to score orderings, you can select an order using entropy of the model's predictions on a probing set: orderings that yield confident, low-entropy, well-spread predictions tend to generalize better (Lu et al., 2022).

This lets you pick a good permutation without a labeled validation set.

def select_order_by_entropy(perms, probe_inputs, build):
    def score(perm):
        ents = [entropy(model_probs(build(perm), x)) for x in probe_inputs]
        return -mean(ents)        # prefer confident, low-entropy orderings
    return max(perms, key=score)

Stabilizing With Calibration

Order sensitivity and recency bias both inflate certain labels. Contextual calibration mitigates this by estimating the model's prior on a content-free input and correcting predicted probabilities, reducing the impact of any single ordering.

Calibration plus a decent ordering yields more stable production behavior than chasing the single best permutation.

p_prior = model_probs(build(perm), content_free='N/A')
def calibrate(probs):
    return normalize(probs / p_prior)  # cancels order-induced label skew

Caching and Order Stability

If you use prompt caching, the prefix (including the demonstration block) must stay byte-stable to hit the cache. Frequently reshuffling examples per request destroys cache reuse and raises cost and latency.

Resolution: fix a single validated order for the static demo block and cache it; reserve dynamic reordering for the retrieved, query-specific tail only.

# Stable cached prefix + dynamic tail
prefix = render(FIXED_ORDERED_DEMOS)   # cache_control on this block
tail   = render(order_by_similarity(q_emb, retrieved))
prompt = prefix + tail + query

An Ordering Strategy Checklist

Practical defaults: balance and alternate labels, place key/hard examples at the edges (avoid the middle), order retrieved examples by ascending similarity so the best sits near the query, measure order variance, and apply calibration to dampen residual skew.

Then lock the order for caching and re-validate whenever the example set changes.

def final_order(demos, q_emb):
    demos = alternate_labels(group(demos), len(demos))
    return order_by_similarity(q_emb, demos)  # validated, then cached

Quick Check

Reason about the position of a critical demonstration.

Recap

Key takeaways:

Example order is a hyperparameter that can move accuracy by several points; search and report variance.
Recency bias makes the final demo most influential; 'lost in the middle' under-weights central examples.
Balance/alternate labels, order retrieved demos by ascending similarity, and consider curriculum ordering.
Select orderings via entropy when labels are scarce, and apply contextual calibration to dampen skew.
Lock a stable order for prompt caching; reorder only the dynamic retrieved tail.

Frequently asked questions

Is the “Example Ordering and Recency” lesson free?

Yes — the full text of “Example Ordering and Recency” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Example Ordering and Recency”?

How example order affects output. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Example Ordering and Recency” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering