AI Prompt Engineering · Lesson

Re-ranking Retrieved Chunks

Cross-encoder re-ranking.

Re-ranking Retrieved Chunks is a free AI Prompt Engineering lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Re-Rank at All

First-stage retrieval (dense or sparse) optimizes for recall at scale: get the gold chunk somewhere in the top 50. It is fast but coarse. A second-stage re-ranker then reorders that shortlist for precision, surfacing the truly relevant chunks to the top.

This retrieve-broadly-then-rerank-precisely pattern is the backbone of advanced RAG.

def two_stage(query, k_retrieve=50, k_final=5):
    candidates = first_stage_retrieve(query, k_retrieve)  # high recall
    reranked = rerank(query, candidates)                  # high precision
    return reranked[:k_final]

Bi-Encoder vs Cross-Encoder

A bi-encoder encodes query and document separately into vectors and compares by cosine; fast and indexable but loses query-document interaction. A cross-encoder feeds the query and a candidate together through the model and outputs a relevance score, capturing fine-grained interaction.

Cross-encoders are far more accurate but cannot be precomputed, so they only run on the shortlist.

# Bi-encoder: score = cos(enc(q), enc(d))     -> precomputable
# Cross-encoder: score = model(q, d) -> 0..1   -> per-pair, no index
def cross_encode(query, doc):
    return cross_encoder.predict([(query, doc)])[0]  # joint attention

A Cross-Encoder Re-Rank Pass

The re-ranker scores each candidate against the query, then sorts descending. Because the cross-encoder attends to query and document jointly, it resolves subtle relevance the bi-encoder missed: negation, exact-match needs, and answer-vs-topic distinctions.

This stage typically lifts answer accuracy more than any other single RAG improvement.

def rerank(query, candidates):
    pairs = [(query, c.text) for c in candidates]
    scores = cross_encoder.predict(pairs)        # batched
    for c, s in zip(candidates, scores):
        c.rerank_score = s
    return sorted(candidates, key=lambda c: c.rerank_score, reverse=True)

LLM-as-Re-Ranker

When no trained cross-encoder fits your domain, an LLM can re-rank. Listwise prompting asks the model to order a list of passages by relevance in one call; pointwise scores each passage independently.

Listwise captures relative comparisons and is token-efficient, but watch for position bias and ensure output parsing is robust to the model omitting or duplicating IDs.

def llm_listwise(query, candidates):
    passages = '\n'.join(
        '[' + str(i) + '] ' + c.text for i, c in enumerate(candidates)
    )
    prompt = (
        'Rank the passages by relevance to the query. '
        'Return only IDs, most relevant first.\nQuery: ' + query +
        '\n' + passages
    )
    order = parse_ids(llm(prompt, temperature=0))
    return [candidates[i] for i in order]

Latency and the Shortlist Size

Re-ranking cost scales with the shortlist size. A cross-encoder on 50 candidates is far cheaper than on 500. Choose the first-stage k large enough to capture the gold chunk (validate recall@k) but small enough to re-rank within your latency budget.

Batch the pair scoring and run it on accelerated hardware; cross-encoders parallelize well across pairs.

def tune_shortlist(eval_set, ks=(20, 50, 100, 200)):
    # find smallest k where recall@k saturates -> rerank fewer pairs
    return {k: (recall_at_k(eval_set, k), rerank_latency(k)) for k in ks}

Hybrid First Stage Plus Re-Rank

The strongest recall comes from a hybrid first stage (dense + BM25 fused), feeding a single deduplicated shortlist into the re-ranker. Dense recovers paraphrases; sparse recovers exact identifiers; the cross-encoder then sorts the union by true relevance.

This combination is robust across query types, from natural-language questions to literal lookups.

def hybrid_then_rerank(query, k_final=6):
    dense = dense_retrieve(query, 50)
    sparse = bm25_retrieve(query, 50)
    fused = dedup(rrf(dense, sparse))     # reciprocal rank fusion
    return rerank(query, fused)[:k_final]

Score Thresholds and Cutoffs

Re-ranker scores are calibratable. Instead of always taking top-n, apply a relevance threshold: keep chunks above a score, and if none qualify, return an honest no-answer. This prevents stuffing the prompt with weakly relevant filler.

Tune the threshold on a validation set to balance answerable coverage against distractor inclusion.

def threshold_select(reranked, tau=0.3, max_n=8):
    kept = [c for c in reranked if c.rerank_score >= tau][:max_n]
    if not kept:
        return None        # signal: no sufficiently relevant context
    return kept

Diversity After Re-Ranking

A pure relevance sort can return several near-duplicate chunks from the same document, wasting the context budget. Apply MMR or per-document caps after re-ranking to ensure the final set covers distinct facets and sources.

This matters for multi-hop questions whose answer spans several documents.

def diversify(reranked, max_per_doc=2, k=6):
    out, per_doc = [], {}
    for c in reranked:
        d = c.meta['doc_id']
        if per_doc.get(d, 0) < max_per_doc:
            out.append(c)
            per_doc[d] = per_doc.get(d, 0) + 1
        if len(out) == k:
            break
    return out

Ordering for the Generator

After selecting the top chunks, place them to exploit attention. Given lost-in-the-middle, put the single highest-scoring chunk at the start or end of the context, not buried among others.

Some pipelines order chunks by ascending relevance so the best sits closest to the question, mirroring few-shot recency strategy.

def order_for_llm(chunks):
    chunks = sorted(chunks, key=lambda c: c.rerank_score)  # ascending
    return chunks                 # most relevant chunk ends up last,
                                  # nearest the trailing question

Evaluating the Re-Ranker

Measure the re-ranker with ranking metrics, primarily NDCG and MRR, on labeled query-chunk relevance, and then the downstream answer accuracy. A re-ranker that improves NDCG but not answers may be reordering chunks the generator already handled.

Always close the loop on end-task quality, not just ranking metrics.

import math

def ndcg_at_k(relevances, k):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(relevances[:k]))
    ideal = sorted(relevances, reverse=True)
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal[:k]))
    return dcg / idcg if idcg else 0.0

A Production Re-Rank Pipeline

End to end: hybrid retrieve a 50-candidate shortlist, dedup, cross-encoder re-rank, apply a score threshold, diversify by document, order for attention, and generate with citations. Gate on the threshold to return honest no-answers.

Cache embeddings and re-ranker scores per (query, chunk) where traffic repeats to cut cost.

def pipeline(query):
    shortlist = hybrid_then_rerank(query, k_final=20)
    kept = threshold_select(shortlist, tau=0.3, max_n=8)
    if kept is None:
        return 'No relevant information found.'
    ctx = order_for_llm(diversify(kept))
    return generate_with_citations(query, ctx)

Quick Check

Choose the right re-ranking architecture.

Recap

Key takeaways:

Retrieve broadly for recall, then re-rank the shortlist for precision.
Cross-encoders score query-document pairs jointly (accurate, not indexable); bi-encoders are fast but coarse.
LLM listwise/pointwise re-ranking is a fallback; watch position bias and parsing.
Tune shortlist size to saturate recall within latency budget; combine with hybrid first-stage retrieval.
Apply score thresholds, diversify by document, order chunks for attention, and evaluate with NDCG plus downstream answer accuracy.

Frequently asked questions

Is the “Re-ranking Retrieved Chunks” lesson free?

Yes — the full text of “Re-ranking Retrieved Chunks” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Re-ranking Retrieved Chunks”?

Cross-encoder re-ranking. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Re-ranking Retrieved Chunks” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering