Text Classification with BERT
HuggingFace transformers, AutoTokenizer, AutoModelForSequenceClassification, fine-tuning.
Text Classification with BERT is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Limits of Static Embeddings
Word2Vec and GloVe give each word one fixed vector. But "bank" means different things in "river bank" and "savings bank". Contextual models like BERT fix this.
What Is BERT
BERT is a transformer model that reads a whole sentence at once and produces context-aware embeddings. Pretrained on huge text, it can be fine-tuned for tasks like classification.
The Hugging Face transformers Library
The transformers library gives easy access to BERT and thousands of pretrained models, with consistent classes for tokenizers and models.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
name = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)AutoTokenizer
AutoTokenizer loads the correct tokenizer for a model. It splits text into subword tokens and maps them to the IDs the model expects.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tokenizer.tokenize("unbelievable"))
# subwords like ["un", "##bel", "##ievable"]AutoModelForSequenceClassification
AutoModelForSequenceClassification loads BERT with a classification head on top, outputting one score per class for the whole sequence.
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased-finetuned-sst-2-english"
)Tokenizing Input
Call the tokenizer on your text with truncation, padding, and return_tensors to get model-ready tensors. Truncation caps long inputs; padding aligns batch lengths.
inputs = tokenizer(
"This movie was fantastic!",
truncation=True,
padding=True,
return_tensors="pt",
)
print(inputs["input_ids"].shape)The Special Tokens
BERT adds special tokens automatically: [CLS] at the start (its hidden state summarizes the sequence) and [SEP] between/after sentences. The classifier reads from the [CLS] representation.
Running the Model
Pass the tokenized inputs to the model to get logits, the raw, unnormalized class scores.
import torch
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
print(logits)Logits to Predictions
Apply softmax to logits for probabilities, then argmax for the predicted class index. Map that index to a label.
import torch
probs = torch.softmax(logits, dim=-1)
pred = torch.argmax(probs, dim=-1).item()
print(model.config.id2label[pred])Fine-Tuning for Your Task
For a custom dataset, fine-tune BERT by training the classification head (and optionally the whole model) on your labeled examples. The Trainer API handles the training loop.
from transformers import Trainer, TrainingArguments
args = TrainingArguments(output_dir="out", num_train_epochs=3)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train()The pipeline Shortcut
For quick inference, the pipeline helper wraps tokenizing, running, and decoding into one call, ideal for prototyping.
from transformers import pipeline
clf = pipeline("sentiment-analysis")
print(clf("I love this product!"))
# [{"label": "POSITIVE", "score": 0.99...}]Quick Check
Test your BERT knowledge.
Recap
Recap: BERT produces context-aware embeddings via transformers. Use AutoTokenizer to tokenize with truncation/padding/return_tensors, and AutoModelForSequenceClassification to get logits. Convert logits with softmax then argmax. Fine-tune with Trainer, or prototype quickly with pipeline.
Frequently asked questions
Is the “Text Classification with BERT” lesson free?
Yes — the full text of “Text Classification with BERT” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Text Classification with BERT”?
HuggingFace transformers, AutoTokenizer, AutoModelForSequenceClassification, fine-tuning. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Text Classification with BERT” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Word2Vec: Skip-gram and CBOW
- GloVe and FastText Embeddings
- Text Classification with BERT
- Semantic Similarity and Sentence Embeddings