Criando um vocabulário

A vocabulary is the full set of unique words your model knows. Each word gets one fixed slot in every document vector. 📖

To build it, gather all words across your documents and keep only the distinct ones, dropping repeats.

Sort the unique words and assign each a number. This word to index map is your vocabulary lookup.

The index tells a word which position it occupies in every vector, so cat always lands in the same column.

If your vocabulary has 5000 words, every document becomes a vector of 5000 numbers, one slot per word.

Once set, the mapping must never change. A stable order guarantees that column 3 means the same word for every document.

A word not in your vocabulary is called out of vocabulary. The simplest choice is to ignore it during counting.

Words that appear once add noise and size. Dropping very rare terms keeps the vocabulary smaller and cleaner.

You can also cut words that appear in nearly every document. These stopwords rarely help tell documents apart.

With the map built, finding a word position is instant. Just index the dictionary by the word you want.

This map is the backbone of vectorization. Next you fill each slot with how often that word appears in a document.

What does a word index represent?

You built a vocabulary of unique words, mapped each to a fixed index, and saw how pruning keeps it lean and useful. 🎉