Dynamic Few-Shot Selection
Retrieving examples per query.
Dynamic Few-Shot Selection is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Static to Dynamic Demos
Static few-shot uses the same examples for every query. Dynamic few-shot retrieves the most relevant demonstrations per query from a pool, conditioning the model on examples that resemble the current input.
This is retrieval-augmented in-context learning: it raises the relevance of demonstrations, which is one of the strongest levers for ICL quality, especially on heterogeneous traffic.
class DynamicSelector:
def __init__(self, pool, embedder, index):
self.pool = pool # candidate demonstrations
self.embed = embedder
self.index = index # ANN index over pool embeddings
def select(self, query, k):
q = self.embed(query)
ids = self.index.search(q, k)
return [self.pool[i] for i in ids]Embedding the Demonstration Pool
Precompute embeddings for every candidate demonstration and store them in an approximate nearest neighbor index (FAISS, HNSW, or a vector DB). At query time, embed the input once and retrieve the top matches.
Choose an embedding model aligned with your task semantics; a generic embedder may cluster on surface features rather than the dimension that predicts the correct label.
import numpy as np
def build_index(pool, embed):
vecs = np.stack([embed(d.input) for d in pool]).astype('float32')
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True) # cosine via dot
index = HNSW(dim=vecs.shape[1])
index.add(vecs)
return indexkNN Prompting
The canonical method (Liu et al., 2022) retrieves the k nearest neighbors of the query from the labeled pool and uses them as demonstrations. Retrieval-based selection consistently beats random selection because relevant demos better locate the task and supply the right label space.
The retrieved labels also act as a soft kNN classifier prior, nudging the model toward neighbors' answers.
def knn_prompt(query, selector, k, build):
demos = selector.select(query, k)
demos = order_by_similarity(embed(query), demos) # most similar last
return build(demos, query)Diversity-Aware Retrieval
Pure top-k can return near-duplicates, wasting context. Apply MMR or clustering over the retrieved candidates to keep relevance while ensuring the chosen demos cover distinct facets of the query.
This matters most for compositional inputs where different sub-aspects each need a representative demonstration.
def diverse_retrieve(query, selector, k, pool_n=30, lam=0.7):
cand = selector.select(query, pool_n)
q = embed(query)
return mmr_against_query(cand, q, k, lam) # relevance + diversityLatency and the Retrieval Budget
Dynamic selection adds an embedding call and an ANN lookup to every request. Budget this: cache query embeddings for repeated inputs, batch retrieval, and keep the index in memory.
For high-QPS systems, the retrieval step must be sub-millisecond; otherwise the relevance gain is eaten by added tail latency.
from functools import lru_cache
@lru_cache(maxsize=50_000)
def cached_embed(text):
return embed(text)
# Plus: warm in-RAM HNSW, batched search, async prefetchCaching Tension With Dynamic Demos
Dynamic demos break prompt-prefix caching because the example block changes per query. Mitigate by keeping a stable cached preamble (instructions plus a few universal exemplars) and appending only the retrieved, query-specific demos after it.
This recovers most caching savings while preserving per-query relevance for the tail.
prompt = (
STATIC_PREAMBLE # cached: instructions + anchor demos
+ render(diverse_retrieve(query, selector, k)) # dynamic tail
+ format_query(query)
)Avoiding Train/Test Leakage
If the query itself is in the pool (common during evaluation), retrieval can return the exact answer, inflating metrics. Always exclude the query and near-identical neighbors above a similarity threshold during evaluation.
In production, deduplicate the pool and guard against echoing a user's own prior input back as a demonstration.
def leak_safe_select(query, selector, k, sim_cap=0.97):
cand = selector.select(query, k + 5)
q = embed(query)
cand = [d for d in cand if cos(q, d.emb) < sim_cap]
return cand[:k]Cold Start and Pool Growth
Early on, the pool is small and retrieval may return weak matches. Bootstrap with a curated static set, then grow the pool from verified production traces, re-embedding and re-indexing on a schedule.
Track per-demo usage and outcome so you can prune low-value or stale examples and keep the index lean.
def maybe_add_to_pool(trace, verified):
if verified and novelty(trace, index) > THRESH:
emb = embed(trace.input)
index.add(emb)
pool.append(Demo(trace.input, trace.output, trace.meta))Selection Beyond Similarity
Nearest-neighbor relevance is a strong default but not always optimal. Advanced selectors weigh informativeness (does the demo resolve the query's ambiguity?), diversity, and label coverage. Some methods learn a selection policy that maximizes downstream accuracy rather than raw similarity.
Frame selection as choosing the demonstration set that most reduces the model's uncertainty on this query.
def select_by_uncertainty_reduction(query, pool, k):
base = entropy(model_probs(build([], query)))
gains = []
for d in pool:
h = entropy(model_probs(build([d], query)))
gains.append((d, base - h)) # info gain per demo
return [d for d, _ in sorted(gains, key=lambda x: -x[1])[:k]]Evaluating a Dynamic Pipeline
Compare dynamic against static and random baselines on held-out, leakage-controlled data. Report accuracy, the distribution of retrieval similarities, end-to-end latency, and cache hit rate.
A dynamic system that wins on accuracy but tanks cache reuse may be net-negative in cost; evaluate the full objective, not just quality.
def eval_pipeline(eval_set):
return {
'acc_dynamic': run(dynamic, eval_set),
'acc_static': run(static, eval_set),
'acc_random': run(random_sel, eval_set),
'p95_latency': latency_p95(),
'cache_hit': cache_hit_rate(),
}Reference Architecture
End to end: a verified demonstration pool, an embedding model, an in-memory ANN index, a selector applying leakage guards, diversity, and similarity ordering, a stable cached preamble, and a feedback loop that grows and prunes the pool.
This architecture turns few-shot prompting into a retrieval system with the operational discipline that implies.
def answer(query):
demos = leak_safe_select(query, selector, k=4)
demos = mmr_against_query(demos, embed(query), 4)
demos = order_by_similarity(embed(query), demos)
prompt = STATIC_PREAMBLE + render(demos) + format_query(query)
out = llm(prompt)
log_for_pool_growth(query, out)
return outQuick Check
Diagnose a misleadingly strong evaluation result.
Recap
Key takeaways:
- Dynamic few-shot retrieves per-query demonstrations, beating random/static by raising relevance.
- Embed the pool into an ANN index; kNN prompting is the strong default, ordered by ascending similarity.
- Add diversity (MMR) and consider informativeness/uncertainty-reduction selectors.
- Guard against retrieval leakage, manage latency, and keep a cached static preamble with a dynamic tail.
- Grow and prune the pool from verified traces; evaluate accuracy, latency, and cache hit together.
Frequently asked questions
Is the “Dynamic Few-Shot Selection” lesson free?
Yes — the full text of “Dynamic Few-Shot Selection” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.
What will I learn in “Dynamic Few-Shot Selection”?
Retrieving examples per query. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Prompt Engineering?
No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Dynamic Few-Shot Selection” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Prompt Engineering lesson?
Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Zero, One, and Few-Shot
- Designing Effective Examples
- Example Ordering and Recency
- Dynamic Few-Shot Selection