Tokenizing for Transformer Models
Subwords, padding, and attention masks.
Tokenizing for Transformer Models is a free NLP Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the NLP Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Tokens Come First
Before a transformer can learn anything, your text must become numbers. That conversion job belongs to the tokenizer.
Match the Model
Always load the tokenizer that was trained with your model. A mismatched vocabulary produces garbage ids the model never saw.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")Subword Pieces
Modern tokenizers split rare words into smaller chunks called subwords, so even unknown text stays representable.
print(tok.tokenize("tokenization"))Why Subwords Win
Subwords keep the vocabulary small while still handling typos and new words. The model rarely meets a true unknown token.
From Tokens to Ids
Each subword maps to an integer id. Calling the tokenizer on text returns those ids ready for the model.
enc = tok("Fine-tuning is fun")
print(enc["input_ids"])Special Tokens
Tokenizers add markers like CLS and SEP so the model knows where a sequence starts and ends. These are the special tokens.
Padding to Equal Length
Batches need same-length rows, so shorter texts get filler tokens. This step is called padding.
tok(texts, padding=True)Truncation for Long Text
Models cap input length, so very long text is cut to fit. Enabling truncation keeps every example within the limit.
tok(texts, truncation=True, max_length=128)The Attention Mask
An attention mask marks real tokens as 1 and padding as 0, so the model ignores the filler positions.
print(enc["attention_mask"])Return Tensors
Ask the tokenizer for framework tensors directly so the output drops straight into training without extra conversion.
tok("hello", return_tensors="pt")Decode Back to Text
To read predictions, run ids through decode and the tokenizer rebuilds the original text, special tokens stripped.
print(tok.decode(enc["input_ids"]))Quick Check
What is the job of the attention mask during batching?
Recap
Tokenizers turn text into subword ids, add special tokens, then handle padding, truncation, and the attention mask for clean batches. ✅
Frequently asked questions
Is the “Tokenizing for Transformer Models” lesson free?
Yes — the full text of “Tokenizing for Transformer Models” is free to read here on the web, and the NLP Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the NLP Academy course, upgrade to CoddyKit PRO.
What will I learn in “Tokenizing for Transformer Models”?
Subwords, padding, and attention masks. You practise NLP Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start NLP Academy?
No prior experience is required. NLP Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Tokenizing for Transformer Models” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this NLP Academy lesson?
Yes. Every NLP Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- The Transformers Library Tour
- Tokenizing for Transformer Models
- Fine-Tuning With the Trainer API
- Evaluating and Saving Your Model