Analyzers, Tokenizers, and the Inverted Index
Learn how Elasticsearch turns text into searchable tokens using analyzers and stores them in an inverted index.
Analyzers, Tokenizers, and the Inverted Index is a free Elasticsearch & Full Text Search Systems lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Elasticsearch & Full Text Search Systems learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Text to Tokens
Before text is searchable, an analyzer breaks it into tokens and stores them in an inverted index. That's the first step of every search.
The Inverted Index
An inverted index maps each token to the documents that contain it — like a book's index — making full-text lookups extremely fast.
Anatomy of an Analyzer
An analyzer runs three stages in order: character filters clean raw text, a tokenizer splits it into tokens, and token filters reshape them.
The Standard Analyzer
The default standard analyzer splits on word boundaries and lowercases, so Quick Brown Fox! becomes quick, brown, fox.
Testing with _analyze
The _analyze API shows exactly which tokens an analyzer produces from your text — run it to see the output.
POST /_analyze
{
"analyzer": "standard",
"text": "The Quick Brown Foxes"
}Stemming
A stemming filter reduces words to their root, so running, runs, and ran all map to run — and searching one finds the others.
Stop Words
A stop filter removes low-value words like the, a, and is, shrinking the index and improving relevance.
Custom Analyzer
Define a custom analyzer in index settings by combining a tokenizer with filters — the code wires up lowercase, stop words, and stemming.
PUT /articles
{
"settings": { "analysis": { "analyzer": {
"my_english": {
"tokenizer": "standard",
"filter": ["lowercase", "english_stop", "english_stemmer"]
}}}}
}text vs keyword
A text field is analyzed for full-text search; a keyword field stays one exact token for filtering, sorting, and aggregations.
Index vs Search Time
Analysis runs at index time when storing a document and again at search time on the query — usually the same analyzer, so tokens match.
Multi-field Mapping
A handy trick: map a string as text for search and add a .keyword sub-field for exact matches and aggregations at once.
"title": { "type": "text",
"fields": { "raw": { "type": "keyword" } } }Quick Check
Why are text and keyword fields treated differently?
Recap
Recap: analyzers (filters, tokenizer, filters) build the inverted index, stemming and stop words refine it, and text vs keyword shapes how fields behave.
Frequently asked questions
Is the “Analyzers, Tokenizers, and the Inverted Index” lesson free?
Yes — the full text of “Analyzers, Tokenizers, and the Inverted Index” is free to read here on the web, and the Elasticsearch & Full Text Search Systems course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Elasticsearch & Full Text Search Systems course, upgrade to CoddyKit PRO.
What will I learn in “Analyzers, Tokenizers, and the Inverted Index”?
Learn how Elasticsearch turns text into searchable tokens using analyzers and stores them in an inverted index. You practise Elasticsearch & Full Text Search Systems with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Elasticsearch & Full Text Search Systems?
No prior experience is required. Elasticsearch & Full Text Search Systems on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Analyzers, Tokenizers, and the Inverted Index” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Elasticsearch & Full Text Search Systems lesson?
Yes. Every Elasticsearch & Full Text Search Systems lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- What is Full-Text Search?
- Elasticsearch Core Concepts
- Setting Up Your First Cluster
- Analyzers, Tokenizers, and the Inverted Index