0Pricing
AI Engineering Academy · Lesson

Contextual Compression and Relevance Filtering

Apply contextual compression to strip irrelevant sentences from retrieved chunks before feeding them to the LLM, reducing noise and saving tokens.

Contextual Compression and Relevance Filtering is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Problem with Noisy Retrieved Chunks

Retrieved chunks often contain mixed relevance content. A 500-token chunk about database indexing might answer the first two sentences of the query but contain six sentences of unrelated material about backup procedures. Sending this entire chunk to the LLM wastes tokens, reduces the signal-to-noise ratio, and can cause the model to generate an answer grounded in the irrelevant portion rather than the relevant sentences.

What Is Contextual Compression?

Contextual compression is a post-retrieval step that takes each retrieved chunk and extracts only the sentences relevant to the query before passing the chunk to the LLM. The original chunk is compressed to its most relevant parts, reducing token usage and improving answer quality. LangChain's ContextualCompressionRetriever wraps any retriever with a compressor component that performs this extraction.

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain_openai import ChatOpenAI

# LLMChainExtractor uses an LLM to extract the relevant portion
llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
compressor = LLMChainExtractor.from_llm(llm)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever,
)

results = compression_retriever.invoke('how does HNSW indexing work?')
for doc in results:
    print(len(doc.page_content), 'chars:', doc.page_content[:150])

LLMChainFilter: Relevance Filtering

Instead of extracting sentences from chunks, LLMChainFilter makes a binary decision: is this chunk relevant to the query or not? Irrelevant chunks are dropped entirely before reaching the LLM. This is cheaper than extraction (shorter LLM call) and useful when chunks are short and coherent enough that partial extraction does not help. Typically 20-40 percent of initially retrieved chunks are filtered out.

from langchain.retrievers.document_compressors import LLMChainFilter

filter_compressor = LLMChainFilter.from_llm(
    llm=ChatOpenAI(model='gpt-4o-mini', temperature=0)
)

filtering_retriever = ContextualCompressionRetriever(
    base_compressor=filter_compressor,
    base_retriever=base_retriever,
)

# Fetch 10 docs, filter drops irrelevant ones
results = filtering_retriever.invoke('what are the pgvector distance operators?')
print(f'{len(results)} chunks passed the relevance filter')

Embeddings-Based Relevance Filtering

Using an LLM for filtering adds latency and cost. A cheaper alternative is embedding-based filtering, which computes the cosine similarity between the query embedding and each chunk's embedding and drops chunks below a similarity threshold. This is deterministic, fast, and free — it reuses embeddings already computed during retrieval without an additional LLM call.

from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
embeddings_filter = EmbeddingsFilter(
    embeddings=embeddings,
    similarity_threshold=0.76,  # cosine similarity threshold
)

embedding_retriever = ContextualCompressionRetriever(
    base_compressor=embeddings_filter,
    base_retriever=base_retriever,
)

results = embedding_retriever.invoke('BM25 hyperparameter tuning')
print(f'Filtered to {len(results)} relevant chunks')

Chaining Multiple Compressors

You can chain multiple compressors in sequence using DocumentCompressorPipeline. A common pattern is to first apply a fast embeddings-based filter to remove clearly irrelevant chunks, then apply sentence splitting to break long chunks into sentences, and finally apply LLM extraction to pull the most relevant sentences. This layered approach balances cost and accuracy.

from langchain.retrievers.document_compressors import DocumentCompressorPipeline
from langchain_community.document_transformers import EmbeddingsRedundantFilter
from langchain_text_splitters import CharacterTextSplitter

# Step 1: split chunks into individual sentences
sentence_splitter = CharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=0,
    separator='. ',
)

# Step 2: remove redundant sentences via embeddings
redundant_filter = EmbeddingsRedundantFilter(embeddings=OpenAIEmbeddings())

# Step 3: keep only relevant sentences
relevance_filter = EmbeddingsFilter(embeddings=OpenAIEmbeddings(), similarity_threshold=0.76)

pipeline = DocumentCompressorPipeline(
    transformers=[sentence_splitter, redundant_filter, relevance_filter]
)

piped_retriever = ContextualCompressionRetriever(
    base_compressor=pipeline,
    base_retriever=base_retriever,
)

Removing Redundant Chunks

When multiple retrieved chunks say essentially the same thing, sending all of them to the LLM wastes tokens without adding information. EmbeddingsRedundantFilter removes near-duplicate chunks by computing pairwise cosine similarity and dropping chunks that are too similar to already selected ones. This is particularly valuable when your corpus has many overlapping chunks due to high overlap settings during ingestion.

from langchain_community.document_transformers import EmbeddingsRedundantFilter
from langchain_core.documents import Document

redundant_filter = EmbeddingsRedundantFilter(
    embeddings=OpenAIEmbeddings(),
    similarity_threshold=0.95,  # treat docs with >95% cosine sim as duplicates
)

# Simulate near-duplicate documents
docs = [
    Document(page_content='pgvector is a PostgreSQL extension for vector similarity search'),
    Document(page_content='pgvector extends PostgreSQL to support vector similarity search'),  # near duplicate
    Document(page_content='HNSW and IVFFlat are the two index types in pgvector'),
]

filtered = redundant_filter.transform_documents(docs, query='')
print(f'Kept {len(filtered)} of {len(docs)} documents after deduplication')

Sentence-Level Relevance Extraction

For long documents where relevant content is scattered throughout, sentence-level extraction with a cross-encoder provides the highest precision. Score every sentence in the retrieved chunk against the query, keep only sentences above a threshold, and reconstruct a compressed document. This approach typically reduces context size by 40-70 percent while preserving all relevant sentences.

from sentence_transformers import CrossEncoder
import re

sentence_reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def compress_with_cross_encoder(
    query: str, chunk: str, min_score: float = 0.0
) -> str:
    sentences = re.split(r'(?<=[.!?]) +', chunk)
    if len(sentences) <= 1:
        return chunk

    pairs = [[query, s] for s in sentences]
    scores = sentence_reranker.predict(pairs)

    relevant = [
        sentence
        for sentence, score in zip(sentences, scores)
        if float(score) >= min_score
    ]
    return ' '.join(relevant) if relevant else chunk

Token Budget Management via Compression

Contextual compression is also an effective token budget management tool. If you want to pass 5 documents to a model with a limited context window, compressing each from 500 tokens to 100 relevant tokens lets you fit 5x more information. This is especially valuable when working with smaller, faster models like GPT-4o-mini that have shorter context windows and stricter latency requirements.

def compress_to_budget(
    query: str,
    docs: list[str],
    total_token_budget: int = 2000,
    tokens_per_char: float = 0.25,
) -> list[str]:
    compressed = []
    used_tokens = 0

    for doc in docs:
        compressed_doc = compress_with_cross_encoder(query, doc)
        doc_tokens = int(len(compressed_doc) * tokens_per_char)

        if used_tokens + doc_tokens <= total_token_budget:
            compressed.append(compressed_doc)
            used_tokens += doc_tokens
        else:
            break  # stop when budget is exhausted

    print(f'Using {used_tokens} tokens across {len(compressed)} docs')
    return compressed

Measuring Compression Quality

Compression introduces a risk: you might accidentally remove a sentence that is critical for answering the query. Measure compression quality by comparing faithfulness scores before and after compression using RAGAS or a custom LLM judge that checks whether the compressed context still supports the correct answer. If faithfulness drops after compression, lower the threshold or switch to extraction rather than filtering.

def measure_compression_faithfulness(query, original_docs, compressed_docs, llm):
    # Use LLM to check if answer from compressed context matches
    # answer from full context
    def generate_answer(docs, q):
        context = '\n'.join(docs)
        resp = llm.invoke(f'Answer from context:\n{context}\nQ: {q}')
        return resp.content

    full_answer = generate_answer([d.page_content for d in original_docs], query)
    comp_answer = generate_answer(compressed_docs, query)

    # Check semantic similarity between answers
    vecs = [embed(full_answer), embed(comp_answer)]
    sim = cosine_similarity(vecs[0], vecs[1])
    print(f'Answer similarity after compression: {sim:.3f}')
    return sim

When to Skip Compression

Contextual compression is not always beneficial. For short coherent chunks (under 200 tokens), the entire chunk is usually relevant and compression adds only latency. For technical documentation where the answer depends on all parts of a procedure (e.g., a numbered step sequence), filtering individual sentences can remove critical steps. Apply compression judiciously and always validate it improves answer quality for your specific use case.

Production-Ready Compression Pipeline

A robust production compression pipeline combines embedding-based filtering (fast, cheap), redundancy removal (avoid repeating the same information), and optional cross-encoder sentence extraction (accurate but slower). Run the fast stages first and apply the expensive LLM-based stage only for high-value queries or when the fast stages leave too many chunks. This adaptive approach optimizes both cost and quality.

class AdaptiveCompressor:
    def __init__(self, embeddings, cross_encoder=None, threshold=0.76):
        self.emb_filter = EmbeddingsFilter(embeddings=embeddings,
                                           similarity_threshold=threshold)
        self.dedup = EmbeddingsRedundantFilter(embeddings=embeddings)
        self.cross_encoder = cross_encoder

    def compress(self, query: str, docs, use_cross_encoder: bool = False):
        # Stage 1: embedding filter
        docs = self.emb_filter.compress_documents(docs, query=query)
        # Stage 2: deduplication
        docs = self.dedup.transform_documents(docs, query=query)
        # Stage 3: optional sentence-level extraction
        if use_cross_encoder and self.cross_encoder:
            docs = [
                type(d)(page_content=compress_with_cross_encoder(
                    query, d.page_content
                ), metadata=d.metadata)
                for d in docs
            ]
        return docs

Quick Check

Test your understanding of contextual compression from this lesson.

Lesson Recap

In this lesson you learned: contextual compression reduces noise by extracting or filtering irrelevant content from retrieved chunks before they reach the LLM, DocumentCompressorPipeline chains multiple compressors (filter, deduplicate, extract) in sequence, and embedding-based filtering provides a fast cheap alternative to LLM-based compression for most scenarios. Compression helps manage token budgets and improves answer quality. Next up we measure the actual impact of re-ranking on retrieval quality.

Frequently asked questions

Is the “Contextual Compression and Relevance Filtering” lesson free?

Yes — the full text of “Contextual Compression and Relevance Filtering” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Contextual Compression and Relevance Filtering”?

Apply contextual compression to strip irrelevant sentences from retrieved chunks before feeding them to the LLM, reducing noise and saving tokens. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Contextual Compression and Relevance Filtering” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Why Two-Stage Retrieval Works
  2. Cross-Encoder Re-ranking with Cohere and BGE
  3. Contextual Compression and Relevance Filtering
  4. Measuring the Impact of Re-ranking
← Back to AI Engineering Academy