Analyzers, Tokenizer und der Inverted Index

Before text is searchable, an analyzer breaks it into tokens and stores them in an inverted index. That's the first step of every search.

An inverted index maps each token to the documents that contain it — like a book's index — making full-text lookups extremely fast.

An analyzer runs three stages in order: character filters clean raw text, a tokenizer splits it into tokens, and token filters reshape them.

The default standard analyzer splits on word boundaries and lowercases, so Quick Brown Fox! becomes quick, brown, fox.

The _analyze API shows exactly which tokens an analyzer produces from your text — run it to see the output.

A stemming filter reduces words to their root, so running, runs, and ran all map to run — and searching one finds the others.

A stop filter removes low-value words like the, a, and is, shrinking the index and improving relevance.

Define a custom analyzer in index settings by combining a tokenizer with filters — the code wires up lowercase, stop words, and stemming.

A text field is analyzed for full-text search; a keyword field stays one exact token for filtering, sorting, and aggregations.

Analysis runs at index time when storing a document and again at search time on the query — usually the same analyzer, so tokens match.

A handy trick: map a string as text for search and add a .keyword sub-field for exact matches and aggregations at once.

Recap: analyzers (filters, tokenizer, filters) build the inverted index, stemming and stop words refine it, and text vs keyword shapes how fields behave.