Beyond Naive RAG
Limitations of basic retrieval.
Beyond Naive RAG is a free AI Prompt Engineering lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Naive RAG Does
Naive RAG is the baseline: chunk documents, embed them, store vectors, embed the query, retrieve top-k by cosine similarity, stuff the chunks into the prompt, and generate. It is a strong starting point and fails in predictable ways at scale.
Understanding those failure modes is the prerequisite for the advanced techniques (re-ranking, compression, query rewriting) covered in this course.
def naive_rag(query, k=5):
q = embed(query)
chunks = vector_store.search(q, k) # top-k by cosine
context = '\n\n'.join(c.text for c in chunks)
return llm('Context:\n' + context + '\n\nQ: ' + query)Retrieval Recall vs Precision
Naive top-k optimizes raw vector similarity, which conflates relevance with surface semantic closeness. You face a tension: a small k risks missing the answer (low recall); a large k floods the context with distractors (low precision).
The embedding similarity that drives retrieval is a coarse proxy for true relevance, and it is the root of several downstream problems.
# The core dilemma
# small k -> may miss the gold chunk (recall problem)
# large k -> distractors crowd context (precision + cost problem)
# Advanced RAG decouples 'retrieve many' from 'use few'The Embedding Mismatch Problem
Queries and documents often live in different linguistic registers: a short question versus a long declarative passage. Bi-encoder embeddings may place a relevant answer far from the question because they were phrased differently (the vocabulary-mismatch problem).
This drives techniques like query rewriting and HyDE that reshape the query to match the document space before retrieval.
# Query: 'how do I revoke a token?'
# Doc: 'Token invalidation is performed via the /sessions endpoint.'
# Lexically and semantically distant -> bi-encoder may miss it
sim = cos(embed('how do I revoke a token?'),
embed('Token invalidation via /sessions')) # may be lowLost in the Middle
Even when the right chunk is retrieved, stuffing many chunks triggers the lost-in-the-middle effect: the model under-attends content placed in the center of a long context. A correct chunk buried at rank 3 of 10 may be effectively ignored.
This motivates re-ranking (put the best chunk where the model attends) and compression (shrink the context so nothing is buried).
# Retrieval rank != attention rank
# Place the highest-relevance chunk at the START or END,
# never stranded in the middle of a large concatenation.Distractor Sensitivity
LLMs are sensitive to irrelevant context. Adding plausible-but-wrong chunks can pull the answer off course, even when the correct chunk is also present. More retrieved context is not monotonically better.
This is why precision matters: a tight, re-ranked, compressed context often beats a large dump of loosely related chunks.
# Empirically: appending a single highly-similar but WRONG chunk
# can flip a previously-correct answer. RAG quality depends on
# keeping distractors OUT, not just getting the gold chunk IN.Chunking Pathologies
Fixed-size chunking splits ideas mid-sentence, separates a claim from its evidence, and strips structural context (which section, which document). A chunk that reads coherently in isolation may be useless or misleading without its surroundings.
Advanced pipelines use structure-aware chunking, overlap, parent-document expansion, and metadata to preserve meaning.
def structure_aware_chunks(doc, max_tokens=400, overlap=50):
sections = split_by_headings(doc) # respect document structure
chunks = []
for sec in sections:
for c in sliding_window(sec.text, max_tokens, overlap):
chunks.append(Chunk(c, meta={'section': sec.title}))
return chunksSemantic-Only Retrieval Gaps
Pure dense retrieval misses exact-match needs: identifiers, error codes, rare proper nouns, API names. These are precisely where users expect literal precision. Hybrid retrieval fuses dense (semantic) and sparse (BM25/keyword) signals to cover both.
Reciprocal rank fusion is a simple, robust way to merge the two ranked lists without tuning a weight.
def rrf(dense_ranks, sparse_ranks, k0=60):
scores = {}
for ranks in (dense_ranks, sparse_ranks):
for rank, doc_id in enumerate(ranks):
scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k0 + rank)
return sorted(scores, key=scores.get, reverse=True)Stale and Unverifiable Context
Naive RAG has no notion of freshness or provenance. It may retrieve outdated docs and offers no built-in way to attribute claims to sources, undermining trust and making hallucination hard to detect.
Advanced systems attach metadata (timestamp, source, version), filter on it, and require the generator to cite chunk IDs so answers are verifiable.
def filtered_retrieve(q, after_date):
cands = vector_store.search(embed(q), k=50)
fresh = [c for c in cands if c.meta['date'] >= after_date]
return fresh # then re-rank; generator must cite c.idNo Feedback, No Adaptation
Naive RAG retrieves blindly: it cannot tell when retrieval failed, cannot decide that no retrieval is needed, and cannot iterate. Advanced patterns add a relevance check, conditional retrieval, and multi-step (agentic) retrieval that reformulates the query when results look weak.
The pipeline becomes a loop with self-assessment rather than a single forward pass.
def adaptive_rag(q):
chunks = retrieve(q)
if relevance_score(q, chunks) < 0.4:
q2 = rewrite_query(q) # reformulate and retry
chunks = retrieve(q2)
if relevance_score(q, chunks) < 0.4:
return 'I could not find this in the sources.'
return generate(q, chunks)The Advanced RAG Stack
Putting the failures together, an advanced pipeline layers: structure-aware chunking with metadata, hybrid retrieval with high recall, a cross-encoder re-ranker for precision, context compression to fit and focus, query rewriting / HyDE to fix mismatch, and a relevance gate with citations.
The next lessons build each layer. The throughline: retrieve broadly, then aggressively filter and refine.
def advanced_rag(q):
cands = hybrid_retrieve(rewrite_query(q), k=50) # high recall
top = rerank(q, cands)[:8] # precision
ctx = compress(q, top) # focus + fit
return generate_with_citations(q, ctx) # verifiableMeasure Before You Optimize
Diagnose which failure you actually have before adding machinery. Measure retrieval recall@k (is the gold chunk retrieved at all) separately from answer accuracy (does the generator use it). A recall problem and a precision problem demand different fixes.
Instrument both halves; do not bolt on a re-ranker when your real issue is chunking or query mismatch.
def diagnose(eval_set):
return {
'recall@5': recall_at_k(eval_set, k=5), # retrieval health
'recall@50': recall_at_k(eval_set, k=50), # ceiling with rerank
'answer_acc': answer_accuracy(eval_set), # generation health
}Quick Check
Diagnose a RAG failure mode.
Recap
Key takeaways:
- Naive RAG (chunk, embed, top-k, stuff, generate) is a strong baseline with predictable failures.
- Bi-encoder similarity is a coarse relevance proxy; query/document register mismatch hurts recall.
- More context is not better: distractor sensitivity and lost-in-the-middle degrade answers as k grows.
- Chunking pathologies, semantic-only gaps, staleness, and lack of feedback all limit naive RAG.
- Advanced RAG retrieves broadly then filters: hybrid retrieval, re-ranking, compression, query rewriting, and relevance gating. Measure recall and answer accuracy separately before optimizing.
Frequently asked questions
Is the “Beyond Naive RAG” lesson free?
Yes — the full text of “Beyond Naive RAG” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.
What will I learn in “Beyond Naive RAG”?
Limitations of basic retrieval. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Prompt Engineering?
No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Beyond Naive RAG” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Prompt Engineering lesson?
Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.