แบ่งโทเค็นและสร้างคลังคำศัพท์

A neural net cannot read raw letters. Before any learning happens, you must turn text into numbers the model can crunch. 🔢

To tokenize means to split text into small pieces called tokens. The simplest choice is to split a sentence on spaces into words.

A token need not be a whole word. It can be a character or a sub-word piece, which helps the model handle rare or unseen words.

A vocabulary is the full set of unique tokens your model knows. You collect every distinct token across your training text.

Each unique token gets one integer id. This mapping from token to id is how words become numbers the network can index.

To encode text, you look up each token in the vocabulary and replace it with its id, producing a list of integers.

Some tokens at test time were never seen in training. Map them to a special unknown token so the model still gets a valid id.

Add special tokens like padding, start, and end markers. They give the model structure beyond the plain words themselves.

Normalizing text by lowercasing and stripping punctuation shrinks the vocabulary so Cat and cat share a single id.

Real corpora have huge vocabularies. Keep only the most frequent tokens to control the vocab size; the rest become unknown.

Once encoded, a sentence is a list of ids you can wrap in a tensor. The pipeline from raw text to model input is complete.

What does building a vocabulary give every unique token?

You split text into tokens, gather the unique ones into a vocabulary, and map each to an integer id so text becomes numbers. ✅