การสร้างแบบจำลองภาษาด้วยการปิดบัง

BERT trains on plain text with no human labels at all. This clever trick is called self-supervised learning.

BERT learns by hiding some words and guessing them back. That game is masked language modeling, or MLM for short.

A random word gets swapped for a special placeholder. Here the model must recover what the mask is covering up.

BERT reads everything around the blank and ranks likely fills. Strong guesses come from using the full context.

To fill a blank well, the model must learn grammar, facts, and word relationships. The blank forces real understanding.

BERT masks about 15 percent of the words in each sentence during training. That ratio balances signal and difficulty.

Unlike left-to-right models, BERT sees words before and after the blank. This bidirectional view makes its guesses far richer.

Original BERT also guessed if one sentence follows another. That extra job is next sentence prediction, teaching it about flow.

This slow, expensive MLM phase is the pre-training step. You download the result and reuse it without redoing the work.

A fill-mask pipeline lets you watch MLM in action and see BERT rank its top predictions for any blank.

After MLM, BERT holds a huge store of language knowledge ready to power your tasks. All of it came from raw text.

What does masked language modeling ask BERT to do?

BERT learns by filling masked blanks using both sides of a sentence. This self-supervised MLM builds deep language knowledge. ✅