Training a Multinomial Naive Bayes Classifier
Learners will fit MultinomialNB on a spam/ham dataset, tune the alpha smoothing parameter, and evaluate with classification_report.
Training a Multinomial Naive Bayes Classifier is a free Machine Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Multinomial Naive Bayes for Text?
Scikit-learn provides three Naive Bayes variants. MultinomialNB is designed for discrete count data — exactly what CountVectorizer produces. It models the probability of each word count given the class. BernoulliNB works with binary presence/absence features and is suited for very short documents. GaussianNB assumes continuous features with Gaussian distributions — appropriate for numerical data, not word counts. For text classification, MultinomialNB is almost always the right choice because word counts are non-negative integers that fit the multinomial distribution assumption perfectly.
from sklearn.naive_bayes import MultinomialNB, BernoulliNB, GaussianNB
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['buy cheap meds now', 'hello friend meeting tomorrow',
'click here for discount', 'project update next week']
labels = [1, 0, 1, 0] # 1=spam, 0=ham
vec = CountVectorizer()
X = vec.fit_transform(corpus)
nb = MultinomialNB()
nb.fit(X, labels)
print('Classes:', nb.classes_)
print('Feature log probs shape:', nb.feature_log_prob_.shape)
# feature_log_prob_[class][feature] = log P(feature | class)Loading and Splitting the 20 Newsgroups Dataset
The 20 Newsgroups dataset is the canonical text classification benchmark. It contains approximately 18,000 newsgroup posts across 20 topics, from 'sci.space' to 'talk.politics.guns'. Scikit-learn provides it via fetch_20newsgroups() with the convenient option to pre-remove headers, footers, and quotes that would make the task artificially easy. We select a binary subset (spam vs. not-spam is the hardest to get in this dataset; instead we compare two newsgroups) and split into train/test.
from sklearn.datasets import fetch_20newsgroups
from sklearn.model_selection import train_test_split
# Load two categories for binary classification
cats = ['sci.space', 'rec.sport.hockey']
train = fetch_20newsgroups(subset='train', categories=cats,
remove=('headers', 'footers', 'quotes'))
test = fetch_20newsgroups(subset='test', categories=cats,
remove=('headers', 'footers', 'quotes'))
print('Train samples:', len(train.data))
print('Test samples:', len(test.data))
print('Categories:', train.target_names)
print('Sample (first 200 chars):', train.data[0][:200])Building a Complete Text Classification Pipeline
The standard pattern for text classification is: CountVectorizer → MultinomialNB → Pipeline. Wrapping them in a Pipeline ensures that the vectoriser is fitted only on training data even during cross-validation. The Pipeline accepts raw text strings as input and produces predictions directly, hiding all intermediate transformation steps. This is the cleanest, most deployment-ready structure for NLP pipelines in scikit-learn.
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, classification_report
pipe = Pipeline([
('vec', CountVectorizer(stop_words='english')),
('clf', MultinomialNB())
])
pipe.fit(train.data, train.target)
preds = pipe.predict(test.data)
print('Accuracy:', accuracy_score(test.target, preds).round(3))
print(classification_report(test.target, preds,
target_names=train.target_names))The Alpha Smoothing Parameter
The most important hyperparameter in MultinomialNB is alpha — the Laplace/Lidstone smoothing parameter. Without smoothing, any word that appears in test data but was never seen in training data for a given class gets P(word|class)=0, making the entire log-probability -infinity. Adding alpha prevents zero probabilities: P(word|class) = (count + alpha) / (total_counts + alpha * vocab_size). The default alpha=1.0 (Laplace smoothing) is usually a good starting point; smaller values (0.01-0.1) can improve performance when data is abundant.
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import cross_val_score
alpha_values = [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0]
best_alpha, best_score = 1.0, 0
for alpha in alpha_values:
pipe = Pipeline([
('vec', CountVectorizer(stop_words='english')),
('nb', MultinomialNB(alpha=alpha))
])
score = cross_val_score(pipe, train.data, train.target, cv=5).mean()
print(f'alpha={alpha:.3f}: CV accuracy = {score:.4f}')
if score > best_score:
best_score, best_alpha = score, alpha
print('Best alpha:', best_alpha)Inspecting Learned Probabilities
After training, MultinomialNB stores feature_log_prob_ — the log-probability of each word given each class. Inspecting the words with the highest log-probability per class reveals what the model learned. For a sci.space vs. rec.sport.hockey classifier, you expect 'nasa', 'orbit', 'launch' to be top features for space, and 'hockey', 'team', 'goal' for hockey. If unexpected words appear (like 'the'), your preprocessing (stopwords, min_df) needs adjustment.
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
import numpy as np
pipe = Pipeline([
('vec', CountVectorizer(stop_words='english')),
('nb', MultinomialNB(alpha=0.1))
])
pipe.fit(train.data, train.target)
vec = pipe.named_steps['vec']
nb = pipe.named_steps['nb']
feature_names = vec.get_feature_names_out()
for class_idx, class_name in enumerate(train.target_names):
top_idx = np.argsort(nb.feature_log_prob_[class_idx])[-10:]
top_words = [feature_names[i] for i in top_idx]
print(f'Top words for {class_name}:', top_words)Classification Report: Precision, Recall, and F1
classification_report provides per-class precision (of all predicted positives, how many are correct), recall (of all actual positives, how many did we find), and F1-score (harmonic mean of precision and recall). For text classification, accuracy alone can be misleading if one class is more common. F1-score is the standard metric for text classification benchmarks because it balances precision and recall. The 'macro avg' averages metrics equally across classes; 'weighted avg' weights by class support.
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np
preds = pipe.predict(test.data)
print('=== Classification Report ===')
print(classification_report(test.target, preds,
target_names=train.target_names))
print('=== Confusion Matrix ===')
cm = confusion_matrix(test.target, preds)
print(cm)
print('Rows = actual class, Columns = predicted class')
print('Diagonal = correct predictions')Predicting on New Text Examples
Once trained, the Pipeline accepts raw text strings for prediction — no manual vectorisation required. predict() returns class labels; predict_proba() returns class probabilities (applying softmax over log posteriors internally). The probabilities reflect the model's confidence. For spam filtering, you might set a stricter threshold (0.9 for spam) to reduce false positives — preferring to let some spam through rather than accidentally blocking legitimate email.
new_emails = [
'NASA launches new Mars mission with rocket booster',
'The team scored three goals in the hockey championship',
'Buy cheap meds online click here for discount offer'
]
predictions = pipe.predict(new_emails)
probabilities = pipe.predict_proba(new_emails)
for text, pred, prob in zip(new_emails, predictions, probabilities):
class_name = train.target_names[pred]
confidence = max(prob)
print(f'Pred: {class_name} ({confidence:.2%})')
print(f'Text: {text[:60]}...')
print()TF-IDF vs Count Vectoriser Comparison
Comparing CountVectorizer and TfidfVectorizer with MultinomialNB requires special care: MultinomialNB expects non-negative count-like inputs, and TF-IDF values are fractional — technically valid but the model may interpret them differently. TfidfVectorizer with sublinear_tf=True and use_idf=True often gives better results with Naive Bayes. For logistic regression and SVM, TF-IDF is almost always better. For Naive Bayes specifically, raw counts with Laplace smoothing are the theoretically correct approach.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
for VecClass, name in [
(CountVectorizer, 'CountVec'),
(TfidfVectorizer, 'TfidfVec')
]:
pipe = Pipeline([
('vec', VecClass(stop_words='english')),
('nb', MultinomialNB(alpha=0.1))
])
score = cross_val_score(pipe, train.data, train.target, cv=5).mean()
print(f'{name} + MultinomialNB: {score:.4f}')Multi-Class Text Classification with 20 Categories
Naive Bayes naturally handles multi-class classification without modification. Using all 20 newsgroups categories, the Pipeline trains one set of word probabilities per class and predicts the class with the highest posterior. Multi-class F1 macro scores around 0.75-0.85 are typical for Naive Bayes on 20 Newsgroups — remarkably competitive for such a simple model. Confusion between similar categories (e.g., 'sci.space' and 'sci.astronomy') is expected and can be diagnosed by inspecting the off-diagonal confusion matrix entries.
from sklearn.datasets import fetch_20newsgroups
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import f1_score
# All 20 categories
train_all = fetch_20newsgroups(subset='train', remove=('headers','footers','quotes'))
test_all = fetch_20newsgroups(subset='test', remove=('headers','footers','quotes'))
pipe = Pipeline([
('vec', TfidfVectorizer(stop_words='english', max_features=50000)),
('nb', MultinomialNB(alpha=0.05))
])
pipe.fit(train_all.data, train_all.target)
preds = pipe.predict(test_all.data)
f1 = f1_score(test_all.target, preds, average='macro')
print(f'20-class Macro F1: {f1:.3f}')ComplementNB: Better for Imbalanced Text Classes
ComplementNB is a variant of MultinomialNB that corrects for the assumption that classes are equally distributed. Instead of estimating P(word|class), it estimates P(word|NOT class) — the probability of the word in all other classes combined — and uses the complement. This approach is more robust when class sizes are imbalanced (common in real email datasets where spam may be 10% of traffic). ComplementNB consistently outperforms MultinomialNB on text benchmarks and is the recommended default for text classification in scikit-learn's own documentation.
from sklearn.naive_bayes import MultinomialNB, ComplementNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
for NB_Class in [MultinomialNB, ComplementNB]:
pipe = Pipeline([
('vec', TfidfVectorizer(stop_words='english')),
('nb', NB_Class(alpha=0.1))
])
score = cross_val_score(
pipe, train_all.data, train_all.target, cv=5
).mean()
print(f'{NB_Class.__name__}: {score:.4f}')
# ComplementNB typically 1-3% betterGaussianNB for Continuous Features
When features are continuous (not word counts), use GaussianNB. It assumes each feature follows a Gaussian (normal) distribution within each class and estimates mean and variance per feature-class combination from training data. Prediction computes the log-likelihood under each class's Gaussian model and adds the log prior. GaussianNB works well when features are genuinely Gaussian-distributed per class, such as sensor readings or physical measurements. It is fast, requires no hyperparameter tuning (no alpha), and is a reasonable first baseline for any continuous-feature classification problem.
from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
gnb = GaussianNB()
gnb.fit(X_tr, y_tr)
print('GaussianNB test accuracy:', gnb.score(X_te, y_te).round(3))
print('CV accuracy:', cross_val_score(gnb, X, y, cv=10).mean().round(3))
print()
print('Learned means per class and feature:')
print(gnb.theta_.round(2))Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: MultinomialNB models word count probabilities from CountVectorizer output, the alpha smoothing parameter prevents zero probabilities for unseen words, and wrapping vectoriser and classifier in a Pipeline creates a clean, deployable text classification system. Next up we explore Laplace smoothing in detail and the zero-probability problem it solves.
Frequently asked questions
Is the “Training a Multinomial Naive Bayes Classifier” lesson free?
Yes — the full text of “Training a Multinomial Naive Bayes Classifier” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Training a Multinomial Naive Bayes Classifier”?
Learners will fit MultinomialNB on a spam/ham dataset, tune the alpha smoothing parameter, and evaluate with classification_report. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Training a Multinomial Naive Bayes Classifier” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Bayes' Theorem in Plain Language
- Bag of Words: CountVectorizer and TfidfVectorizer
- Training a Multinomial Naive Bayes Classifier
- Laplace Smoothing and Zero-Probability Problem