Cleaning and Deduplicating Source Data
Learn to clean noisy documents and remove duplicate content before ingestion so your RAG index stays small, accurate, and free of conflicting answers.
Cleaning and Deduplicating Source Data is a free LLM Apps in Production (RAG + Vector DB + Caching) lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the LLM Apps in Production (RAG + Vector DB + Caching) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Garbage In, Garbage Out
RAG quality is capped by the quality of what you ingest. Boilerplate, HTML tags, duplicate pages, and broken encoding all pollute retrieval.
Cleaning and deduplication happen before chunking and embedding.
Common Noise Sources
Typical junk found in raw documents:
- Navigation menus, headers, footers
- Cookie banners and ads
- Repeated legal disclaimers
- Mojibake from bad encoding
- Excess whitespace and control chars
Basic Text Normalization
Normalize whitespace and strip control characters first.
import re
def clean(text):
text = re.sub(r'[\t\r]+', ' ', text)
text = re.sub(r' {2,}', ' ', text)
text = re.sub(r'\n{3,}', '\n\n', text)
return text.strip()
print(clean('Hello world\n\n\n\nbye'))Stripping Boilerplate
Remove repeated boilerplate that appears on many pages. A simple approach: collect lines that repeat across documents and drop them.
- Footers, copyright lines
- Share-this widgets
- Identical navigation blocks
Fixing Encoding Issues
Mojibake like 'caf\u00c3\u00a9' instead of 'caf\u00e9' confuses embeddings. Detect the source encoding and decode consistently to UTF-8 before storage.
Exact Duplicate Detection
The cheapest dedup: hash the normalized text and drop exact repeats.
import hashlib
seen = set()
def is_dup(text):
h = hashlib.sha256(text.encode()).hexdigest()
if h in seen:
return True
seen.add(h)
return False
print(is_dup('a'))
print(is_dup('a'))Near-Duplicate Detection
Exact hashing misses pages that differ by a date or a word. Use near-duplicate techniques:
- MinHash + Jaccard similarity
- SimHash fingerprints
- Embedding cosine similarity above a threshold
Jaccard Similarity
A quick token-set overlap score to flag near-duplicates.
def jaccard(a, b):
sa, sb = set(a.split()), set(b.split())
return len(sa & sb) / len(sa | sb)
print(round(jaccard('the cat sat', 'the cat ran'), 2))Why Duplicates Hurt RAG
Duplicate chunks waste index space and skew retrieval: the top-k results fill up with copies of the same passage, crowding out diverse evidence. Conflicting near-duplicates (old vs new policy) can even produce contradictory answers.
Building a Cleaning Pipeline
Chain the steps in order: normalize -> fix encoding -> strip boilerplate -> exact dedup -> near dedup. Log how much was removed so you can audit aggressive filters.
Idempotent Re-ingestion
When documents are re-ingested, use a stable content hash as the record key so updates replace the old version instead of creating duplicates. This keeps the index clean over time.
Quick Check
Test your understanding of deduplication.
Recap
You learned to prepare clean source data: normalize text, fix encoding, strip boilerplate, then remove both exact and near-duplicates. Use stable content hashes for idempotent re-ingestion. Cleaner inputs mean a smaller index and more accurate, non-contradictory retrieval.
Frequently asked questions
Is the “Cleaning and Deduplicating Source Data” lesson free?
Yes — the full text of “Cleaning and Deduplicating Source Data” is free to read here on the web, and the LLM Apps in Production (RAG + Vector DB + Caching) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the LLM Apps in Production (RAG + Vector DB + Caching) course, upgrade to CoddyKit PRO.
What will I learn in “Cleaning and Deduplicating Source Data”?
Learn to clean noisy documents and remove duplicate content before ingestion so your RAG index stays small, accurate, and free of conflicting answers. You practise LLM Apps in Production (RAG + Vector DB + Caching) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start LLM Apps in Production (RAG + Vector DB + Caching)?
No prior experience is required. LLM Apps in Production (RAG + Vector DB + Caching) on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Cleaning and Deduplicating Source Data” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this LLM Apps in Production (RAG + Vector DB + Caching) lesson?
Yes. Every LLM Apps in Production (RAG + Vector DB + Caching) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Loading Diverse Document Formats
- Context-Aware Chunking Strategies
- Metadata Management and Filtering
- Cleaning and Deduplicating Source Data