Tokenize and Build a Vocabulary
Map text to integer ids.
Tokenize and Build a Vocabulary is a free Deep Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Networks Only Eat Numbers
A neural net cannot read raw letters. Before any learning happens, you must turn text into numbers the model can crunch. 🔢
First Step: Tokenize
To tokenize means to split text into small pieces called tokens. The simplest choice is to split a sentence on spaces into words.
text = "I love deep learning"
tokens = text.split()
# ["I", "love", "deep", "learning"]Tokens Can Be Smaller
A token need not be a whole word. It can be a character or a sub-word piece, which helps the model handle rare or unseen words.
Build a Vocabulary
A vocabulary is the full set of unique tokens your model knows. You collect every distinct token across your training text.
Give Each Token an Id
Each unique token gets one integer id. This mapping from token to id is how words become numbers the network can index.
vocab = {"i": 0, "love": 1, "deep": 2, "learning": 3}Encode a Sentence
To encode text, you look up each token in the vocabulary and replace it with its id, producing a list of integers.
ids = [vocab[t] for t in ["i", "love", "deep"]]
# [0, 1, 2]Handle Unknown Words
Some tokens at test time were never seen in training. Map them to a special unknown token so the model still gets a valid id.
unk_id = vocab.get("dragons", vocab["<unk>"])Special Tokens Help
Add special tokens like padding, start, and end markers. They give the model structure beyond the plain words themselves.
Lowercase and Clean
Normalizing text by lowercasing and stripping punctuation shrinks the vocabulary so Cat and cat share a single id.
Limit the Vocabulary Size
Real corpora have huge vocabularies. Keep only the most frequent tokens to control the vocab size; the rest become unknown.
Now Text Is a Tensor
Once encoded, a sentence is a list of ids you can wrap in a tensor. The pipeline from raw text to model input is complete.
import torch
ids = torch.tensor([0, 1, 2, 3])Quick Check
What does building a vocabulary give every unique token?
Recap
You split text into tokens, gather the unique ones into a vocabulary, and map each to an integer id so text becomes numbers. ✅
Frequently asked questions
Is the “Tokenize and Build a Vocabulary” lesson free?
Yes — the full text of “Tokenize and Build a Vocabulary” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Tokenize and Build a Vocabulary”?
Map text to integer ids. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Tokenize and Build a Vocabulary” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Tokenize and Build a Vocabulary
- nn.Embedding: Learnable Word Vectors
- Why Embeddings Capture Meaning
- Train a Text Classifier