AI Agents · Lesson

Ranking and Filtering Search Results

Scoring relevance, deduplication, and selecting the best results for context.

Ranking and Filtering Search Results is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Ranking and Filtering Matters

A search API returns 5-10 results, but not all are equally relevant, reliable, or useful for the agent's task. Raw results fed directly to the LLM waste context tokens and can introduce noise or misinformation.

Ranking and filtering improves the signal-to-noise ratio before results reach the LLM.

Relevance Scoring with BM25

BM25 (Best Match 25) is a classical text ranking algorithm that scores documents by keyword overlap with the query. It works well for lexical matching — when the query and document share the same words.

Install with pip install rank-bm25.

from rank_bm25 import BM25Okapi

def rank_with_bm25(query, results):
    # Tokenize: lowercase and split into words
    tokenized_results = [
        r['content'].lower().split()
        for r in results
    ]
    bm25 = BM25Okapi(tokenized_results)

    query_tokens = query.lower().split()
    scores = bm25.get_scores(query_tokens)

    # Sort results by score descending
    ranked = sorted(
        zip(scores, results),
        key=lambda x: x[0],
        reverse=True
    )
    return [(score, result) for score, result in ranked]

Relevance Scoring with Embeddings

BM25 only matches exact words. Embedding similarity captures semantic meaning — so 'Python web development' and 'building websites with Django' score high similarity even with different words.

Use cosine similarity between query embedding and result embeddings.

import numpy as np
import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def embed(text):
    resp = client.embeddings.create(
        model='text-embedding-3-small',
        input=text[:8000]
    )
    return np.array(resp.data[0].embedding)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def rank_by_embedding(query, results):
    q_emb = embed(query)
    scored = []
    for r in results:
        r_emb = embed(r['content'][:1000])
        score = cosine_similarity(q_emb, r_emb)
        scored.append((score, r))
    return sorted(scored, key=lambda x: x[0], reverse=True)

Hybrid Ranking: BM25 + Embeddings

BM25 and embedding scoring capture different aspects of relevance. Hybrid ranking combines both scores with a weighted average, getting the best of both lexical and semantic matching.

def hybrid_rank(query, results, bm25_weight=0.4, embed_weight=0.6):
    # Get BM25 scores (normalized 0-1)
    bm25_scored = rank_with_bm25(query, results)
    max_bm25 = max(s for s, _ in bm25_scored) or 1
    bm25_norm = {r['url']: s / max_bm25 for s, r in bm25_scored}

    # Get embedding scores
    embed_scored = rank_by_embedding(query, results)
    embed_norm = {r['url']: s for s, r in embed_scored}

    # Combine
    combined = []
    for r in results:
        url = r['url']
        score = (bm25_weight * bm25_norm.get(url, 0) +
                 embed_weight * embed_norm.get(url, 0))
        combined.append((score, r))

    return sorted(combined, key=lambda x: x[0], reverse=True)

Deduplication by URL

Search results often contain near-duplicates: the same article from multiple syndicated sources, or the same page with different URL parameters. Deduplication removes these before passing results to the LLM.

from urllib.parse import urlparse, urlunparse

def normalize_url(url):
    parsed = urlparse(url)
    # Remove query params and fragment (tracking params, etc.)
    clean = parsed._replace(query='', fragment='')
    return urlunparse(clean).rstrip('/')

def deduplicate_results(results):
    seen_urls = set()
    unique = []
    for r in results:
        url = normalize_url(r.get('url', ''))
        if url not in seen_urls:
            seen_urls.add(url)
            unique.append(r)
    return unique

# Also deduplicate by content similarity (near-duplicate detection)
def deduplicate_by_content(results, min_unique_ratio=0.7):
    unique = [results[0]] if results else []
    for candidate in results[1:]:
        cand_words = set(candidate['content'].lower().split())
        is_duplicate = False
        for kept in unique:
            kept_words = set(kept['content'].lower().split())
            overlap = len(cand_words & kept_words) / max(len(cand_words), 1)
            if overlap > (1 - min_unique_ratio):
                is_duplicate = True
                break
        if not is_duplicate:
            unique.append(candidate)
    return unique

if __name__ == '__main__':
    demo_results = [
        {'url': 'https://example.com/a?utm_source=x'},
        {'url': 'https://example.com/a'},
        {'url': 'https://example.com/b'},
    ]
    unique = deduplicate_results(demo_results)
    print(f'{len(demo_results)} results -> {len(unique)} unique')
    for r in unique:
        print(' -', r['url'])

Domain Quality Scoring

A result from docs.python.org is more trustworthy than one from a random blog. Assign quality multipliers to domain tiers and factor them into the final ranking.

DOMAIN_QUALITY = {
    # Tier 1 — authoritative (1.3x boost)
    'docs.python.org': 1.3,
    'developer.mozilla.org': 1.3,
    'arxiv.org': 1.3,
    'github.com': 1.2,
    'stackoverflow.com': 1.2,
    # Tier 2 — good (1.0x, no change)
    # Tier 3 — low quality (penalty)
    'pinterest.com': 0.3,
    'quora.com': 0.5,
    'wikihow.com': 0.6
}

def get_domain_multiplier(url):
    from urllib.parse import urlparse
    domain = urlparse(url).netloc.lower().replace('www.', '')
    return DOMAIN_QUALITY.get(domain, 1.0)  # default: no change

def apply_domain_boost(scored_results):
    boosted = []
    for score, r in scored_results:
        multiplier = get_domain_multiplier(r.get('url', ''))
        boosted.append((score * multiplier, r))
    return sorted(boosted, key=lambda x: x[0], reverse=True)

if __name__ == '__main__':
    scored = [(1.0, {'url': 'https://pinterest.com/x'}), (1.0, {'url': 'https://docs.python.org/x'})]
    for score, r in apply_domain_boost(scored):
        print(f"{r['url']}: boosted score {score:.2f}")

Filtering Low-Quality Results

Some results are structurally low quality regardless of their domain: too short to be useful, contain mostly navigation text, or are from login-walled pages. Filter these out before ranking.

MIN_CONTENT_LENGTH = 200  # characters

LOW_QUALITY_SIGNALS = [
    'sign in to view',
    'please log in',
    'subscribe to read',
    '404 not found',
    'access denied',
    'this content is for members only'
]

def is_quality_result(result):
    content = result.get('content', '')

    # Too short
    if len(content) < MIN_CONTENT_LENGTH:
        return False

    # Paywall / access barrier detected
    content_lower = content.lower()
    for signal in LOW_QUALITY_SIGNALS:
        if signal in content_lower:
            return False

    return True

def filter_results(results):
    return [r for r in results if is_quality_result(r)]

if __name__ == '__main__':
    demo_results = [
        {'content': 'Please log in to view this article which has plenty of extra padding text here.'},
        {'content': 'A' * 250},
    ]
    kept = filter_results(demo_results)
    print(f'{len(demo_results)} results -> {len(kept)} passed quality filter')

Result Truncation for Context Budget

Even after filtering, you may have 5 high-quality results with 600 characters each — that's 3,000 characters total. Decide how many results fit within your LLM's context budget and truncate accordingly.

MAX_CONTEXT_CHARS = 4000
MAX_SNIPPET_CHARS = 600

def truncate_for_context(ranked_results, budget=MAX_CONTEXT_CHARS):
    selected = []
    used_chars = 0

    for score, result in ranked_results:
        content = result.get('content', '')[:MAX_SNIPPET_CHARS]
        entry = f"Source: {result['title']}\nURL: {result['url']}\nContent: {content}"
        entry_len = len(entry)

        if used_chars + entry_len > budget:
            break

        selected.append(result)
        used_chars += entry_len

    return selected

ranked_results = [
    (0.9, {'title': 'Doc A', 'url': 'http://a', 'content': 'x' * 800}),
    (0.7, {'title': 'Doc B', 'url': 'http://b', 'content': 'y' * 800}),
]
budget = 1000
selected = truncate_for_context(ranked_results, budget=budget)
print(f'Selected {len(selected)} results within {budget}-char budget')

Formatting Results for the LLM Prompt

After ranking, deduplicating, and truncating, format the results as a numbered list in the LLM prompt. Numbered sources make it easy for the model to cite them in its answer.

def format_results_for_prompt(results):
    lines = ['Here are relevant search results:\n']
    for i, r in enumerate(results, 1):
        lines.append(f'[{i}] {r["title"]}')
        lines.append(f'    URL: {r["url"]}')
        lines.append(f'    {r.get("content", "")[:400]}')
        lines.append('')
    lines.append('Use these sources to answer the question. Cite sources as [1], [2], etc.')
    return '\n'.join(lines)

# Usage in agent prompt
formatted = format_results_for_prompt(selected_results)
response = llm_call(
    system='You are a research assistant.',
    user=f'{formatted}\n\nQuestion: {user_question}'
)

Caching Search Results

The same query may be repeated across sessions or agent loops. Cache search results with a short TTL (e.g., 1 hour) to reduce API costs and improve response time for repeated queries.

import hashlib
import time

search_cache = {}  # In production: use Redis or disk cache
CACHE_TTL = 3600  # 1 hour

def cached_search(query, **kwargs):
    cache_key = hashlib.md5(query.encode()).hexdigest()
    entry = search_cache.get(cache_key)

    if entry and (time.time() - entry['ts']) < CACHE_TTL:
        print('Search cache hit')
        return entry['results']

    results = client.search(query=query, **kwargs)
    search_cache[cache_key] = {
        'results': results,
        'ts': time.time()
    }
    return results

Full Filtering and Ranking Pipeline

Chain all steps into a single pipeline function: fetch → filter low quality → deduplicate → rank → apply domain boost → truncate for context → format for prompt.

def search_and_rank(query, max_context_chars=4000):
    # 1. Fetch
    raw = cached_search(query, max_results=8)
    results = raw.get('results', [])

    # 2. Filter quality
    results = filter_results(results)

    # 3. Deduplicate
    results = deduplicate_results(results)

    # 4. Rank (BM25 fast path — save embedding costs)
    scored = rank_with_bm25(query, results)

    # 5. Domain boost
    scored = apply_domain_boost(scored)

    # 6. Truncate to context budget
    selected = truncate_for_context(scored, budget=max_context_chars)

    # 7. Format
    return format_results_for_prompt(selected)

Knowledge Check

What is the main advantage of hybrid ranking (BM25 + embeddings) over using BM25 alone?

Recap: Ranking and Filtering Search Results

Raw search results need processing before reaching the LLM. The pipeline: filter low-quality results (too short, paywalled) → deduplicate by URL and content → rank by BM25 and/or embedding similarity → apply domain quality boosts → truncate to context budget → format as numbered sources.

Cache search results to reduce API costs for repeated queries. Numbered citations in the prompt enable the LLM to attribute claims to specific sources.

Frequently asked questions

Is the “Ranking and Filtering Search Results” lesson free?

Yes — the full text of “Ranking and Filtering Search Results” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Ranking and Filtering Search Results”?

Scoring relevance, deduplication, and selecting the best results for context. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Ranking and Filtering Search Results” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents