Masked Language Modeling
How BERT learns from raw text.
Masked Language Modeling is a free NLP Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the NLP Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Learning Without Labels
BERT trains on plain text with no human labels at all. This clever trick is called self-supervised learning.
The Fill-in-the-Blank Game
BERT learns by hiding some words and guessing them back. That game is masked language modeling, or MLM for short.
Hiding a Word
A random word gets swapped for a special placeholder. Here the model must recover what the mask is covering up.
text = "the cat sat on the [MASK]"
# model should predict: matPredict the Missing Word
BERT reads everything around the blank and ranks likely fills. Strong guesses come from using the full context.
Why Hiding Helps
To fill a blank well, the model must learn grammar, facts, and word relationships. The blank forces real understanding.
The Famous 15 Percent
BERT masks about 15 percent of the words in each sentence during training. That ratio balances signal and difficulty.
Both Sides at Once
Unlike left-to-right models, BERT sees words before and after the blank. This bidirectional view makes its guesses far richer.
A Second Task
Original BERT also guessed if one sentence follows another. That extra job is next sentence prediction, teaching it about flow.
Pre-training, Then Reuse
This slow, expensive MLM phase is the pre-training step. You download the result and reuse it without redoing the work.
Try It in Code
A fill-mask pipeline lets you watch MLM in action and see BERT rank its top predictions for any blank.
from transformers import pipeline
fm = pipeline("fill-mask", model="bert-base-uncased")
fm("Paris is the [MASK] of France.")Knowledge for Free
After MLM, BERT holds a huge store of language knowledge ready to power your tasks. All of it came from raw text.
Quick Check
What does masked language modeling ask BERT to do?
Recap
BERT learns by filling masked blanks using both sides of a sentence. This self-supervised MLM builds deep language knowledge. ✅
Frequently asked questions
Is the “Masked Language Modeling” lesson free?
Yes — the full text of “Masked Language Modeling” is free to read here on the web, and the NLP Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the NLP Academy course, upgrade to CoddyKit PRO.
What will I learn in “Masked Language Modeling”?
How BERT learns from raw text. You practise NLP Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start NLP Academy?
No prior experience is required. NLP Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Masked Language Modeling” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this NLP Academy lesson?
Yes. Every NLP Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Context Changes Word Meaning
- Masked Language Modeling
- Embedding Sentences With BERT
- Picking the Right Pre-trained Model