AI Prompt Engineering · Lesson

Context Compression

Trimming context to what matters.

Context Compression is a free AI Prompt Engineering lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Compress Context

Retrieved chunks are noisy: a relevant chunk may be mostly boilerplate with one load-bearing sentence. Context compression trims retrieved text down to what actually answers the query before it reaches the generator.

Benefits compound: lower token cost, reduced latency, fewer distractors, and relief from lost-in-the-middle by shrinking the context the model must traverse.

def compress(query, chunks):
    # Goal: keep only spans that bear on the query,
    # dropping boilerplate, navigation, and off-topic sentences.
    return [extract_relevant(query, c) for c in chunks]

Extractive vs Abstractive

Extractive compression selects verbatim spans (sentences, passages) relevant to the query, preserving exact wording and provenance. Abstractive compression paraphrases or summarizes, achieving higher compression but risking information loss and introduced errors.

For factual RAG with citations, prefer extractive to keep claims traceable to source; use abstractive only when faithfulness can be verified.

def extractive(query, chunk):
    sents = split_sentences(chunk.text)
    scored = [(s, relevance(query, s)) for s in sents]
    kept = [s for s, r in scored if r > TAU]
    return ' '.join(kept) or top1(scored)   # never return empty

Sentence-Level Filtering

A lightweight, robust technique: score each sentence of each chunk against the query with a small cross-encoder or embedding similarity, and drop low-scoring sentences. This preserves the high-value sentences while discarding filler.

Keep a minimum context per chunk to avoid stripping the surrounding sentence that gives a fact its meaning.

def sentence_filter(query, chunk, keep_ratio=0.4):
    sents = split_sentences(chunk.text)
    scores = embed_sim_batch(query, sents)
    n_keep = max(1, int(len(sents) * keep_ratio))
    idx = sorted(range(len(sents)), key=lambda i: -scores[i])[:n_keep]
    return ' '.join(sents[i] for i in sorted(idx))  # keep original order

LLM-Based Contextual Compression

An LLM can compress per chunk: prompt it to extract only the parts of a passage relevant to the query, returning the chunk verbatim-trimmed or an empty marker if nothing is relevant.

This is the LangChain ContextualCompressionRetriever pattern. It is more accurate than embedding filters but adds an LLM call per chunk, so reserve it for high-stakes pipelines or batch it.

def llm_compress(query, chunk):
    prompt = (
        'Extract ONLY sentences from the passage relevant to the query. '
        'If none are relevant, output NONE. Do not paraphrase.\n'
        'Query: ' + query + '\nPassage: ' + chunk.text
    )
    out = llm(prompt, temperature=0).strip()
    return None if out == 'NONE' else out

Token-Level Pruning (LLMLingua)

Methods like LLMLingua compress at the token level using a small model to estimate token informativeness, dropping low-information tokens. They can achieve large compression ratios while preserving the signal the big model needs.

The tradeoff: the compressed text becomes less human-readable, so this suits machine-only context, not user-facing display.

def token_prune(context, target_ratio=0.3):
    # small LM estimates per-token information (perplexity-based);
    # drop the least informative tokens down to target_ratio length
    scores = small_lm_token_importance(context)
    return keep_top_fraction(context, scores, target_ratio)

Query-Aware vs Query-Agnostic

Query-aware compression keeps what is relevant to this query and yields the tightest context, but must run per request. Query-agnostic compression (precomputed chunk summaries) is cheaper at request time but cannot focus on the specific question.

A hybrid stores condensed chunk versions offline and applies light query-aware trimming online.

# Offline: precompute a dense summary per chunk (query-agnostic)
chunk.summary = summarize(chunk.text)
# Online: query-aware trim over summaries (cheap) then verify on full
ctx = [sentence_filter(query, Chunk(c.summary)) for c in top_chunks]

Guarding Against Information Loss

Aggressive compression can delete the one clause that mattered. Guard with a recall check: verify the compressed context still entails the answer, or keep a fallback to the uncompressed chunk when the compressor returns suspiciously little.

Never let compression drop a chunk to empty when the re-ranker rated it highly relevant; that signals a compressor failure, not irrelevance.

def safe_compress(query, chunk):
    out = llm_compress(query, chunk)
    if out is None and chunk.rerank_score > 0.7:
        return chunk.text          # trust re-ranker over compressor
    return out or chunk.text

Preserving Provenance

Compression must keep source attribution intact. Tag each retained span with its chunk and document ID so the generator can cite it. If you compress away the metadata, you lose verifiability and the ability to detect hallucination.

Carry IDs through the pipeline as structured fields, never as free text that compression might strip.

def compress_with_ids(query, chunks):
    out = []
    for c in chunks:
        span = safe_compress(query, c)
        out.append({'id': c.id, 'doc': c.meta['doc_id'], 'text': span})
    return out   # generator cites id; provenance preserved

Compression and the Token Budget

Compression lets you retrieve more candidates for recall while keeping the final prompt small. Set a token budget for the context window and greedily pack the highest-value compressed spans until the budget is reached.

This decouples how much you retrieve from how much you spend on generation, a key efficiency lever.

def pack(spans, budget_tokens, tok):
    spans = sorted(spans, key=lambda s: -s['score'])
    used, packed = 0, []
    for s in spans:
        t = tok(s['text'])
        if used + t > budget_tokens:
            continue
        packed.append(s); used += t
    return packed

Measuring Compression Quality

Track three numbers: compression ratio (tokens saved), answer accuracy (did quality hold), and faithfulness (did the compressor avoid altering facts). The goal is the highest ratio that does not move answer accuracy.

Sweep compression aggressiveness and pick the Pareto point that meets your cost target without quality loss.

def eval_compression(eval_set, levels):
    return {
        lvl: {
            'ratio': mean_ratio(eval_set, lvl),
            'acc':   answer_accuracy(eval_set, lvl),
            'faith': faithfulness(eval_set, lvl),
        } for lvl in levels
    }

A Compression-Aware Pipeline

End to end: retrieve broadly, re-rank, compress each surviving chunk (extractive or LLM-based) with provenance, guard against over-trimming, pack to a token budget, and generate with citations.

Compression is the layer that makes high-recall retrieval affordable and keeps the generator focused on what matters.

def pipeline(query):
    top = rerank(query, hybrid_retrieve(query, 50))[:12]
    spans = compress_with_ids(query, top)
    spans = [s for s in spans if s['text']]
    packed = pack(spans, budget_tokens=2000, tok=count_tokens)
    return generate_with_citations(query, packed)

Quick Check

Choose the right compression approach for a factual, cited RAG system.

Recap

Key takeaways:

Compression trims retrieved chunks to query-relevant content, cutting cost, latency, and distractors.
Prefer extractive (verbatim, traceable) over abstractive for cited factual RAG; token-level pruning suits machine-only context.
Query-aware compression is tightest but per-request; combine with offline summaries for efficiency.
Guard against information loss and preserve chunk/document provenance for citations.
Use compression to retrieve broadly yet keep prompts small; tune for max ratio without losing answer accuracy.

Frequently asked questions

Is the “Context Compression” lesson free?

Yes — the full text of “Context Compression” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Context Compression”?

Trimming context to what matters. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Context Compression” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering