Machine Learning Academy · درس

تنعيم لابلاس ومشكلة الاحتمال الصفري

أعد إنتاج فشل الاحتمال الصفري للكلمات غير المرئية، وتعرّف إلى كيفية منع تنعيم لابلاس للنموذج من إسناد احتمال صفري

تنعيم لابلاس ومشكلة الاحتمال الصفري درس مجاني في Machine Learning Academy على CoddyKit. هذا هو الدرس 4 من أصل 4. يمكنك قراءة الدرس كاملاً أدناه مجاناً — ثم تمرن عليه مباشرة في المتصفح باستخدام محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7. هذا الدرس جزء من مسار التعلم في Machine Learning Academy، وتقدمك يتزامن عبر الويب وتطبيق CoddyKit. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.

بعض أجزاء هذا الدرس لم تُترجم بعد وتظهر باللغة الإنجليزية.

The Zero-Probability Catastrophe

Naive Bayes computes the posterior probability of a class by multiplying the likelihoods of all features: P(class|features) ∝ P(class) * product of P(feature_i | class). If any single feature has zero probability given a class — because it never appeared in training data for that class — the entire product is zero, regardless of all other features. This means a single unseen word makes the classifier unable to distinguish between classes for that document. This is the zero-probability problem, and it is particularly severe for text data where the vocabulary at test time almost always includes words not seen in training.

import numpy as np

# Training: 'bitcoin' never appeared in spam class
# Test document: 'buy bitcoin now cheap'

words_in_test = ['buy', 'bitcoin', 'now', 'cheap']

# Training probabilities (hypothetical)
P_word_given_spam = {'buy': 0.3, 'bitcoin': 0.0, 'now': 0.2, 'cheap': 0.4}

# Multiply likelihoods
product = 1.0
for word in words_in_test:
    prob = P_word_given_spam.get(word, 0.0)
    product *= prob
    print(f'After {word}: product = {product}')

print('Final P(features|spam) =', product)  # ZERO -- catastrophic!

Why Zero Probability Breaks the Model

When P(features|class) = 0 for multiple classes simultaneously (which happens when unseen words exist), the classifier cannot distinguish them — all posterior probabilities are 0. When only some classes have zero probability, the classifier is forced toward the remaining classes, which may be wrong. This is not just a numerical inconvenience — it is a fundamental model failure. In log-space, a zero probability becomes negative infinity: log(0) = -infinity. Summing with finite values still gives -infinity, so the log-probability is completely dominated by this single zero, ignoring all other evidence.

import numpy as np

# In log-space: log(0) = -inf destroys the sum
log_probs = [np.log(0.3), np.log(0.0), np.log(0.2), np.log(0.4)]

for word, lp in zip(['buy', 'bitcoin', 'now', 'cheap'], log_probs):
    print(f'log P({word}|spam) = {lp}')

log_posterior_spam = sum(log_probs)
print(f'\nLog P(spam|doc) = {log_posterior_spam}')  # -inf
print('Prediction is dominated by the single zero probability!')

Laplace Smoothing: Adding Pseudocounts

Laplace smoothing (also called additive smoothing or add-one smoothing) solves the zero-probability problem by adding a small constant alpha to every word count before computing probabilities. The formula becomes: P(word | class) = (count(word, class) + alpha) / (total_words_in_class + alpha * vocab_size). With alpha=1, every word gets at least one 'virtual' occurrence in every class. This guarantees no word ever has zero probability, while having minimal impact on words that were genuinely frequent in training data.

import numpy as np

def laplace_prob(count_word_class, total_words_class, vocab_size, alpha=1.0):
    return (count_word_class + alpha) / (total_words_class + alpha * vocab_size)

# Parameters
total_spam_words = 1000
vocab_size = 5000
alpha = 1.0

# Word seen 50 times in spam
P_buy_spam = laplace_prob(50, total_spam_words, vocab_size, alpha)
print(f'P(buy|spam)     = {P_buy_spam:.6f}')  # High probability

# Word NEVER seen in spam (count=0)
P_bitcoin_spam = laplace_prob(0, total_spam_words, vocab_size, alpha)
print(f'P(bitcoin|spam) = {P_bitcoin_spam:.6f}')  # Small but non-zero
print('Zero-probability problem solved!')

The Effect of Alpha on Probabilities

The alpha parameter controls how much smoothing is applied. Alpha=1 (Laplace) adds one virtual count per word per class. Larger alpha values produce stronger smoothing — word probabilities move closer to the uniform distribution (1/vocab_size). Very small alpha values (0.001) provide minimal smoothing but are numerically stable. The optimal alpha balances between eliminating zeros and not over-smoothing real signals. Cross-validation is the principled way to select alpha — it typically falls between 0.01 and 1.0 for most text classification tasks.

import numpy as np

total_words = 1000
vocab_size  = 5000
count_buy   = 50   # Seen 50 times
count_new   = 0    # Never seen

print('Alpha comparison for two words:')
print(f'{"alpha":>8} | {"P(buy|class)":>15} | {"P(unseen|class)":>18}')
print('-' * 50)
for alpha in [0.001, 0.01, 0.1, 1.0, 10.0]:
    p_buy = (count_buy + alpha) / (total_words + alpha * vocab_size)
    p_new = (count_new + alpha) / (total_words + alpha * vocab_size)
    print(f'{alpha:>8.3f} | {p_buy:>15.6f} | {p_new:>18.6f}')

Lidstone Smoothing: A Generalisation

Lidstone smoothing is the general form of Laplace smoothing where alpha can be any positive value rather than specifically 1.0. When alpha=1, it is Laplace (add-one) smoothing. When alpha < 1, it is sometimes called Jeffreys-Perks smoothing. There is no universally best alpha value — it depends on the vocabulary size, training data size, and test data characteristics. For large training sets (millions of documents), very small alpha values (0.001) work well because most words are seen. For small training sets, larger alpha values prevent overfitting to the observed word counts.

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import fetch_20newsgroups

train = fetch_20newsgroups(subset='train',
                           categories=['sci.space', 'rec.sport.hockey'],
                           remove=('headers', 'footers', 'quotes'))

pipe = Pipeline([
    ('vec', CountVectorizer(stop_words='english')),
    ('nb', MultinomialNB())
])

# GridSearch over alpha (Lidstone smoothing parameter)
grid = GridSearchCV(pipe, {'nb__alpha': [0.001, 0.01, 0.1, 0.5, 1.0, 5.0]},
                    cv=5)
grid.fit(train.data, train.target)
print('Best alpha:', grid.best_params_['nb__alpha'])
print('Best CV accuracy:', grid.best_score_.round(4))

Demonstrating the Fix: Before and After Smoothing

Here is a concrete comparison showing how smoothing prevents the zero-probability catastrophe. Without smoothing, a document containing one unseen word gets posterior probability 0 for the affected class. With smoothing, the unseen word receives a small but positive probability, preserving the contribution of all other features. The predicted class does not change when the unseen word is non-discriminative, but the probability is now a meaningful number rather than 0 or -infinity.

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

train_texts = ['buy cheap now offer deal', 'hello friend meeting lunch',
               'discount click buy fast', 'project status update report']
train_labels = [1, 0, 1, 0]

vec = CountVectorizer()
X_train = vec.fit_transform(train_texts)

test_text = ['buy bitcoin now']  # 'bitcoin' unseen in training
X_test = vec.transform(test_text)

for alpha in [0.0, 1e-10, 1.0]:
    nb = MultinomialNB(alpha=alpha if alpha > 0 else 1e-300)
    nb.fit(X_train, train_labels)
    prob = nb.predict_proba(X_test)
    print(f'alpha={alpha}: P(ham)={prob[0][0]:.4f}, P(spam)={prob[0][1]:.4f}')

Smoothing in Practice: Vocabulary Handling

Smoothing also helps with out-of-vocabulary (OOV) words at inference time. Words not in the training vocabulary are ignored by CountVectorizer by default (they receive no column in the feature matrix). However, words that were in the vocabulary but had zero count in one class are handled by smoothing. The two mechanisms work together: CountVectorizer's handle_unknown='ignore' (for unseen words entirely) plus MultinomialNB's alpha (for zero-count words within the vocabulary). A robust text pipeline uses both to handle the inevitable vocabulary drift between training and deployment data.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
import numpy as np

train_texts = ['cat dog bird', 'fish tank water', 'dog cat pet']
train_labels = [0, 1, 0]

vec = CountVectorizer()
X_train = vec.fit_transform(train_texts)
print('Training vocabulary:', list(vec.vocabulary_.keys()))

# Test with fully unseen word 'elephant'
test = ['cat dog elephant']  # 'elephant' not in vocabulary
X_test = vec.transform(test)
print('OOV word silently ignored, only cat+dog encoded')

nb = MultinomialNB(alpha=1.0)
nb.fit(X_train, train_labels)
print('Prediction:', nb.predict(X_test))

Smoothing for Other Naive Bayes Variants

BernoulliNB applies Laplace smoothing similarly: instead of word count probabilities, it smooths binary presence probabilities P(feature=1|class) and P(feature=0|class). GaussianNB uses a different approach — it adds a small fraction of the variance to avoid zero variance on constant features: var_smoothing adds this fraction of the largest variance in the dataset to all variances. Each variant has a smoothing mechanism appropriate to its distribution assumption, but all share the same underlying goal: preventing zero probabilities that would make the model fail.

from sklearn.naive_bayes import BernoulliNB, GaussianNB
import numpy as np

# BernoulliNB with alpha smoothing (same as MultinomialNB)
bnb = BernoulliNB(alpha=1.0)

# GaussianNB with var_smoothing to prevent zero variance
gnb = GaussianNB(var_smoothing=1e-9)  # Default: 1e-9 of max variance

print('BernoulliNB smooths P(feature=1|class) with alpha')
print('GaussianNB smooths variance with var_smoothing')
print('Both prevent zero probabilities in their respective distributions')

# var_smoothing default prevents failure on constant features
import numpy as np
X = np.array([[1,1],[2,2],[3,3],[1,2]])
y = np.array([0,0,1,1])
gnb.fit(X, y)
print('GaussianNB trained successfully even with near-constant features')

Relationship to Regularisation in Other Models

Laplace smoothing in Naive Bayes is conceptually similar to L2 regularisation in logistic regression or weight decay in neural networks. All three mechanisms prevent extreme parameter values by pulling estimates toward a neutral baseline (uniform distribution for Laplace, zero weights for L2). The key difference: Laplace smoothing targets probability estimates and has a clear Bayesian interpretation — it is equivalent to adding alpha imaginary observations of each word to each class. Larger alpha corresponds to a stronger prior belief that all words are equally likely, shrinking estimates toward uniformity.

# Bayesian interpretation of Laplace smoothing:
# Laplace smoothing = Dirichlet prior on word probabilities
# alpha=1 = uniform Dirichlet prior (all words equally likely before data)
# alpha -> 0 = no prior (maximum likelihood, prone to zero probs)
# alpha -> inf = strong prior (all words equally likely, ignores data)

import numpy as np

alphas = [0.001, 0.1, 1.0, 10.0, 100.0]
for alpha in alphas:
    # Word counts: 'buy' seen 10x, 'rocket' seen 0x in spam (1000 words, 5000 vocab)
    p_buy = (10 + alpha) / (1000 + alpha * 5000)
    p_rocket = (0 + alpha) / (1000 + alpha * 5000)
    ratio = p_buy / p_rocket
    print(f'alpha={alpha:6.3f}: P(buy)/P(unseen) = {ratio:.1f}x')

Testing Smoothing on a Real-World Scenario

To conclusively demonstrate smoothing's value, compare models with and without it on a dataset where some test words are genuinely absent from the training data. Artificially create this by training on a small corpus and testing on a larger one. Without smoothing, accuracy plummets because out-of-vocabulary words for certain classes cause zero posteriors. With smoothing, the model gracefully handles new words, focusing on the words it does recognise. This robustness is why Naive Bayes with smoothing remains a strong baseline even decades after its introduction.

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.metrics import accuracy_score

train = fetch_20newsgroups(subset='train',
    categories=['sci.space', 'rec.sport.hockey'],
    remove=('headers','footers','quotes'))
test = fetch_20newsgroups(subset='test',
    categories=['sci.space', 'rec.sport.hockey'],
    remove=('headers','footers','quotes'))

vec = CountVectorizer(stop_words='english')
X_tr = vec.fit_transform(train.data)
X_te = vec.transform(test.data)

for alpha in [1e-10, 0.01, 0.1, 1.0]:
    nb = MultinomialNB(alpha=alpha)
    nb.fit(X_tr, train.target)
    acc = accuracy_score(test.target, nb.predict(X_te))
    print(f'alpha={alpha}: test accuracy = {acc:.4f}')

Smoothing Equivalence to Dirichlet Prior

From a Bayesian perspective, Laplace smoothing is equivalent to placing a symmetric Dirichlet prior on the word probability distribution for each class. The Dirichlet distribution is the conjugate prior for the multinomial distribution, meaning the posterior (given observed word counts) is also Dirichlet, and the MAP (maximum a posteriori) estimate is exactly the Laplace-smoothed probability formula. Alpha is the concentration parameter of the Dirichlet prior: alpha=1 is uniform (all words equally probable a priori); alpha<1 is sparse (most words have near-zero probability a priori); alpha>1 is dense (pushes all words toward equal probability). This Bayesian framing explains why small alpha is appropriate for large corpora and large alpha for small corpora.

import numpy as np

# Dirichlet-Multinomial MAP estimate = Laplace smoothing
# P(word_i | class) = (count_i + alpha) / (N + V * alpha)
# where V = vocabulary size, N = total words seen in class

# For large corpus:
N_large = 1_000_000  # 1 million words in class
V = 50_000           # 50k vocabulary
count_unseen = 0

for alpha in [0.001, 0.01, 0.1, 1.0]:
    p_unseen = (count_unseen + alpha) / (N_large + V * alpha)
    print(f'alpha={alpha:.3f}: P(unseen|class) = {p_unseen:.2e}')

# Very small alpha keeps unseen words near-zero (good for large corpus)
# Large alpha smoothes too aggressively, inflates unseen word probs

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the zero-probability problem arises when unseen words make the entire posterior probability collapse to zero, Laplace smoothing adds alpha pseudocounts to guarantee every word has a positive probability, and the optimal alpha is a hyperparameter tuned via cross-validation that balances between eliminating zeros and not over-smoothing real signals. Next up we explore classification metrics including accuracy, precision, recall, and F1-score.

الأسئلة الشائعة

هل درس «تنعيم لابلاس ومشكلة الاحتمال الصفري» مجاني؟

نعم — نص درس «تنعيم لابلاس ومشكلة الاحتمال الصفري» كامل متاح مجاناً هنا على الويب. لتمرينه بشكل تفاعلي (محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7) وفتح باقي دورة Machine Learning Academy، انتقل إلى CoddyKit PRO. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.

ماذا ستتعلم في «تنعيم لابلاس ومشكلة الاحتمال الصفري»؟

أعد إنتاج فشل الاحتمال الصفري للكلمات غير المرئية، وتعرّف إلى كيفية منع تنعيم لابلاس للنموذج من إسناد احتمال صفري تتمرن على Machine Learning Academy مع أكواد عملية تشغلها مباشرة في المتصفح، ومدرس ذكاء اصطناعي متاح 24/7 يجيب على أسئلتك أثناء عملك.

هل أحتاج إلى خبرة سابقة لأبدأ Machine Learning Academy؟

لا تُشترط خبرة سابقة. Machine Learning Academy على CoddyKit منظم للمبتدئين حتى المتقدمين، لذا يمكنك البدء من هنا أو من البداية والتقدم بسرعتك الخاصة. هذا هو الدرس 4 من أصل 4.

كم من الوقت يستغرق درس «تنعيم لابلاس ومشكلة الاحتمال الصفري»؟

معظم دروس CoddyKit تستغرق حوالي 5–10 دقائق. كل منها موجز وتفاعلي، لذا تحرز تقدماً مستمراً وتستأنف من حيث توقفت عبر الويب والتطبيق.

هل يمكنني كتابة وتشغيل أكواد في درس Machine Learning Academy هذا؟

نعم. كل درس في Machine Learning Academy يتضمن محرر أكواد مدمج، لذا تكتب وتشغل أكواداً حقيقية مباشرة في متصفحك وتحصل على تعليقات فورية من الذكاء الاصطناعي — بدون إعداد محلي.

جميع الدروس في هذه الدورة

نظرية بايز بلغة واضحة
حقيبة الكلمات: CountVectorizer وTfidfVectorizer
تدريب مصنّف Multinomial Naive Bayes
تنعيم لابلاس ومشكلة الاحتمال الصفري

← العودة إلى Machine Learning Academy