Example Ordering and Recency
How example order affects output.
Order Is a Hidden Hyperparameter
The order of few-shot examples can swing accuracy by several percentage points, sometimes turning a near-random prompt into a strong one and vice versa. This volatility exists even when the example set is fixed.
Treat ordering as a first-class hyperparameter to search, not an afterthought. Reporting few-shot results without controlling for order is scientifically unsound.
import itertools
def order_search(demos, eval_set, build, max_perms=24):
perms = list(itertools.permutations(demos))[:max_perms]
scored = [(p, evaluate(build(p), eval_set)) for p in perms]
return max(scored, key=lambda x: x[1])Recency Bias Explained
Decoder-only transformers exhibit a recency bias: tokens nearer the generation point exert disproportionate influence. The last demonstration before the query is the most likely to shape format, label, and tone.
This is partly an artifact of attention patterns and positional encoding. The practical upshot: whatever you place last is what the model most readily imitates.
# Position weight intuition (illustrative, not exact)
# influence(example_i) roughly increases with proximity to the query
# => the FINAL demo disproportionately anchors the next generation
weights = [0.1, 0.15, 0.25, 0.5] # earlier ... latestAll lessons in this course
- Zero, One, and Few-Shot
- Designing Effective Examples
- Example Ordering and Recency
- Dynamic Few-Shot Selection