Embedding Sentences With BERT
Extract contextual vectors in code.
Embedding Sentences With BERT is a free NLP Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the NLP Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Words to Sentences
BERT gives every token its own vector. But often you want one vector for the whole sentence, not each word.
The CLS Token
BERT adds a special token named CLS at the very front of each input. Its output is often used as a sentence summary.
Mean Pooling
Another common trick averages all the token vectors into one. This mean pooling step often beats the raw CLS vector.
Load a Model
Hugging Face makes loading easy: grab a tokenizer and a model with one line each. They form your pipeline for embeddings.
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")Tokenize the Text
The tokenizer turns your sentence into id numbers and an attention mask BERT understands. This is the encoding step.
inputs = tok("I love NLP", return_tensors="pt")Run the Model
Pass the encoded inputs into BERT to get hidden states for every token. These vectors are the raw output you will pool.
out = model(**inputs)
states = out.last_hidden_statePool to One Vector
Average the token states along the sequence to collapse them into a single sentence embedding you can store.
vec = states.mean(dim=1)An Easier Path
The Sentence-Transformers library wraps all of this for you. One call returns a clean sentence vector ready to use. ✨
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("all-MiniLM-L6-v2")
vec = m.encode("I love NLP")Compare Sentences
With two sentence vectors you measure closeness using cosine similarity. Higher scores mean the sentences mean similar things.
What You Can Build
Sentence embeddings power semantic search, clustering, and duplicate detection. They turn meaning into something you can search.
A Note on Quality
Models tuned for sentences, like MiniLM, usually beat plain BERT here. Pick one trained for the task you actually have.
Quick Check
How do you turn many token vectors into one sentence vector?
Recap
Tokenize, run BERT, then pool or use CLS to get one sentence embedding. Sentence-Transformers makes it a single call. ✅
Frequently asked questions
Is the “Embedding Sentences With BERT” lesson free?
Yes — the full text of “Embedding Sentences With BERT” is free to read here on the web, and the NLP Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the NLP Academy course, upgrade to CoddyKit PRO.
What will I learn in “Embedding Sentences With BERT”?
Extract contextual vectors in code. You practise NLP Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start NLP Academy?
No prior experience is required. NLP Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Embedding Sentences With BERT” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this NLP Academy lesson?
Yes. Every NLP Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Context Changes Word Meaning
- Masked Language Modeling
- Embedding Sentences With BERT
- Picking the Right Pre-trained Model