AI Prompt Engineering · Lesson

Query Rewriting and HyDE

Improving retrieval recall.

Query Rewriting and HyDE is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Query Is the Weak Link

Retrieval quality is bounded by the query. Raw user queries are often short, ambiguous, full of pronouns, or phrased unlike the documents. Query transformation reshapes the query before retrieval to raise recall and precision.

It is one of the cheapest, highest-leverage upgrades over naive RAG: fix the query and every downstream stage benefits.

def transform_query(raw, history):
    # Resolve references, expand, decompose, or hypothesize
    # BEFORE embedding and retrieving.
    return rewrite(raw, history)

Query Rewriting

The simplest transform: an LLM rewrites the user query into a clearer, self-contained, retrieval-friendly form. It fixes typos, expands abbreviations, and removes conversational noise.

This is especially important in multi-turn chat where the latest message is unintelligible without context (How about the second one?).

def rewrite_query(raw):
    prompt = (
        'Rewrite the user question into a clear, standalone search '
        'query optimized for document retrieval. Keep key entities.\n'
        'Question: ' + raw
    )
    return llm(prompt, temperature=0).strip()

Conversational Condensation

In chat RAG, condense the dialogue and the new turn into a single standalone question. Without this, pronouns and ellipses make the embedding meaningless and retrieval collapses.

Pass prior turns so the rewriter can resolve it, that, the previous one into explicit entities the retriever can match.

def condense(history, follow_up):
    prompt = (
        'Given the conversation, rewrite the follow-up as a standalone '
        'question with all references resolved.\nConversation:\n' +
        render(history) + '\nFollow-up: ' + follow_up
    )
    return llm(prompt, temperature=0).strip()

Query Expansion

Expansion generates synonyms, related terms, or alternate phrasings to widen lexical and semantic coverage. It combats vocabulary mismatch: the document says invalidate, the user says revoke.

Expand for retrieval, then retrieve with the union; do not show the expanded form to the user. Beware over-expansion, which can pull in off-topic results.

def expand(query, n=3):
    prompt = (
        'List ' + str(n) + ' alternative phrasings of this query using '
        'synonyms and domain terms, one per line.\n' + query
    )
    variants = parse_lines(llm(prompt, temperature=0.5))
    return [query] + variants

Multi-Query Retrieval

Generate several diverse reformulations, retrieve for each, and fuse the result lists (reciprocal rank fusion). Different phrasings surface different relevant chunks; fusion combines their strengths and stabilizes recall.

This trades extra retrieval calls for robustness against any single bad phrasing.

def multi_query(raw, n=4):
    queries = expand(raw, n)
    rankings = [dense_retrieve(q, 30) for q in queries]
    fused = rrf(*rankings)
    return dedup(fused)

Query Decomposition

Complex, multi-hop questions need decomposition into sub-questions, each retrieved separately, then composed. Asking which CEO of the company that acquired X is older than... cannot be answered by one retrieval.

Decompose, retrieve per sub-question, and let the generator combine the gathered evidence.

def decompose_and_retrieve(question):
    subs = parse_lines(llm(
        'Break this into independent sub-questions, one per line.\n' +
        question, temperature=0))
    evidence = {s: rerank(s, dense_retrieve(s, 30))[:3] for s in subs}
    return evidence  # generator synthesizes the final answer

HyDE: The Core Idea

HyDE (Hypothetical Document Embeddings, Gao et al., 2022) flips the problem. Instead of embedding the short query, it asks the LLM to generate a hypothetical answer document, then embeds that and retrieves with it.

The hypothetical document lives in the same register as real documents, so its embedding sits closer to genuine answers, fixing the query-document mismatch.

def hyde(query):
    hypo = llm(
        'Write a short passage that would answer this question, as if '
        'from a reference document.\nQuestion: ' + query,
        temperature=0.3)
    return dense_retrieve_by_vector(embed(hypo), k=30)  # embed the answer

Why HyDE Improves Recall

A question and its answer are phrased differently; a question and a fake-but-plausible answer are phrased similarly to the real answer. HyDE leverages the LLM's generative prior to bridge that gap, even if the hypothetical contains factual errors.

Correctness of the hypothetical does not matter much: it only needs the right vocabulary and structure to land near true documents in embedding space.

# Robustness: average several hypothetical docs to reduce variance
def hyde_avg(query, n=3):
    vecs = [embed(llm(HYDE_PROMPT + query, temperature=0.5))
            for _ in range(n)]
    centroid = sum(vecs) / n
    return dense_retrieve_by_vector(centroid, 30)

HyDE Tradeoffs and Failure Modes

HyDE adds an LLM generation before retrieval, increasing latency and cost, and can hallucinate a hypothetical that drifts off-topic, hurting recall for niche or out-of-distribution queries the model knows nothing about.

Mitigate by combining HyDE-vector retrieval with raw-query retrieval via fusion, so a bad hypothetical cannot fully sink recall.

def hyde_hybrid(query):
    a = dense_retrieve_by_vector(embed(hyde_doc(query)), 30)
    b = dense_retrieve(query, 30)          # raw-query fallback
    return dedup(rrf(a, b))                 # robust to bad hypotheticals

Choosing and Combining Techniques

These transforms are complementary. Use condensation for chat, expansion/multi-query for vocabulary gaps, decomposition for multi-hop, and HyDE for question-document register mismatch. Many production stacks chain condense, then HyDE, then fuse with the raw query, then re-rank.

Each added transform costs an LLM call, so gate them by query type rather than always running all.

def route_transform(query, history, qtype):
    q = condense(history, query) if history else query
    if qtype == 'multi_hop': return decompose_and_retrieve(q)
    if qtype == 'mismatch':  return hyde_hybrid(q)
    if qtype == 'vocab_gap': return multi_query(q)
    return dense_retrieve(q, 30)

Evaluating Transforms

Measure each transform by its lift in retrieval recall@k and final answer accuracy versus the raw query, on a leakage-free set. Track the added latency and LLM cost so you only keep transforms that pay for themselves.

Beware: a transform that improves recall but adds distractors can lower answer accuracy unless paired with re-ranking and compression.

def eval_transform(name, fn, eval_set):
    return {
        'recall@10': recall_at_k(eval_set, retriever=fn, k=10),
        'answer_acc': answer_accuracy(eval_set, retriever=fn),
        'extra_latency_ms': transform_latency(fn),
    }

Quick Check

Reason about why HyDE works despite imperfect hypotheticals.

Recap

Key takeaways:

Retrieval is bounded by the query; transforming it is a cheap, high-leverage RAG upgrade.
Rewriting/condensation makes chat queries standalone; expansion and multi-query fix vocabulary gaps.
Decomposition handles multi-hop questions by retrieving per sub-question.
HyDE embeds a hypothetical answer to bridge question-document register mismatch; correctness of the hypothetical is not required.
Combine transforms by query type, fuse with the raw query for robustness, and evaluate recall, answer accuracy, latency, and cost.

Frequently asked questions

Is the “Query Rewriting and HyDE” lesson free?

Yes — the full text of “Query Rewriting and HyDE” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Query Rewriting and HyDE”?

Improving retrieval recall. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Query Rewriting and HyDE” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering