สร้างโทเคนสำหรับโมเดล Transformer

Before a transformer can learn anything, your text must become numbers. That conversion job belongs to the tokenizer.

Always load the tokenizer that was trained with your model. A mismatched vocabulary produces garbage ids the model never saw.

Modern tokenizers split rare words into smaller chunks called subwords, so even unknown text stays representable.

Subwords keep the vocabulary small while still handling typos and new words. The model rarely meets a true unknown token.

Each subword maps to an integer id. Calling the tokenizer on text returns those ids ready for the model.

Tokenizers add markers like CLS and SEP so the model knows where a sequence starts and ends. These are the special tokens.

Batches need same-length rows, so shorter texts get filler tokens. This step is called padding.

Models cap input length, so very long text is cut to fit. Enabling truncation keeps every example within the limit.

An attention mask marks real tokens as 1 and padding as 0, so the model ignores the filler positions.

Ask the tokenizer for framework tensors directly so the output drops straight into training without extra conversion.

To read predictions, run ids through decode and the tokenizer rebuilds the original text, special tokens stripped.

What is the job of the attention mask during batching?

Tokenizers turn text into subword ids, add special tokens, then handle padding, truncation, and the attention mask for clean batches. ✅