Bayes' Theorem in Plain Language
Learners will work through a concrete medical-test example to build intuition for prior, likelihood, and posterior probabilities without heavy math.
Bayes' Theorem in Plain Language is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is Bayes' Theorem?
Bayes' theorem is a mathematical rule for updating beliefs in light of new evidence. It answers the question: Given that I observed X, how probable is hypothesis H? The formula is: P(H|X) = P(X|H) * P(H) / P(X). In plain English: the posterior probability of H given X equals the likelihood of observing X if H were true, times the prior probability of H, divided by the overall probability of observing X. Bayes' theorem is fundamental to machine learning, statistics, and rational reasoning under uncertainty.
# Bayes' theorem components:
# P(H|X) = Posterior -- What we want to know
# 'Probability of H given we observed X'
# P(X|H) = Likelihood -- How likely is X if H is true?
# P(H) = Prior -- Our belief in H before seeing X
# P(X) = Evidence -- Overall probability of observing X
# Rearranged:
# Posterior = (Likelihood * Prior) / Evidence
print('Posterior = (Likelihood * Prior) / Evidence')A Medical Test Example
Let's build intuition with a classic example. A rare disease affects 1% of the population. A test is 95% accurate: if you have the disease, it tests positive 95% of the time; if you don't, it tests negative 95% of the time (5% false positive rate). If you test positive, what is the probability you actually have the disease? Most people's intuition says 95%, but Bayes' theorem reveals the answer is much lower — because the disease is so rare, most positives are actually false positives.
# Medical test: Bayes' theorem applied
P_disease = 0.01 # Prior: 1% have the disease
P_no_disease = 0.99 # 99% are healthy
P_pos_given_disease = 0.95 # True positive rate (sensitivity)
P_pos_given_no_disease = 0.05 # False positive rate
# P(positive) = total probability of a positive test
P_positive = (P_pos_given_disease * P_disease +
P_pos_given_no_disease * P_no_disease)
# Bayes: P(disease | positive)
P_disease_given_pos = (P_pos_given_disease * P_disease) / P_positive
print(f'P(disease | positive test) = {P_disease_given_pos:.2%}')
# Only ~16%! Not 95% -- the low prior dominatesPrior, Likelihood, and Posterior Explained
The three key terms in Bayes' theorem: Prior P(H) is your belief about H before seeing any evidence — in the medical example, the 1% disease prevalence. Likelihood P(X|H) is how well the evidence X is explained by hypothesis H — the 95% true positive rate. Posterior P(H|X) is your updated belief after incorporating the evidence. The posterior from one observation becomes the prior for the next — Bayes' theorem describes a continuous learning process where beliefs are updated as evidence accumulates.
# Sequential Bayesian updating
# Start with 1% prior, observe 3 positive tests
prior = 0.01
P_pos_given_disease = 0.95
P_pos_given_no_disease = 0.05
for test_num in range(1, 4):
likelihood_pos = P_pos_given_disease
P_positive = likelihood_pos * prior + P_pos_given_no_disease * (1 - prior)
posterior = (likelihood_pos * prior) / P_positive
print(f'After test {test_num}: P(disease) = {posterior:.3%}')
prior = posterior # Posterior becomes new prior
# Three positive tests raise probability substantiallyBayes' Theorem for Classification
In machine learning classification, Bayes' theorem gives us the probability of each class given the observed features: P(class | features) ∝ P(features | class) * P(class). We predict the class with the highest posterior probability. P(class) is estimated from class frequencies in training data (prior). P(features | class) is estimated from the feature distribution within each class (likelihood). Naive Bayes simplifies the likelihood calculation by assuming features are independent given the class — a 'naive' assumption that often works surprisingly well in practice.
# Classification with Bayes' theorem
# Predict P(spam | email_features) vs P(ham | email_features)
# Prior (from training data)
P_spam = 0.3 # 30% of emails are spam
P_ham = 0.7 # 70% are ham
# Likelihood: P(features | class) from training
# (simplified: single word 'offer' seen)
P_offer_given_spam = 0.6 # 'offer' appears in 60% of spam
P_offer_given_ham = 0.1 # 'offer' appears in 10% of ham
# Unnormalised posteriors (ignore P(offer) -- same denominator)
posterior_spam = P_offer_given_spam * P_spam # 0.18
posterior_ham = P_offer_given_ham * P_ham # 0.07
print('Unnormalised: spam=', posterior_spam, 'ham=', posterior_ham)
print('Predicted class: spam' if posterior_spam > posterior_ham else 'ham')The Law of Total Probability
The denominator in Bayes' theorem, P(X), is often computed using the law of total probability: P(X) = sum over all classes c of P(X|c) * P(c). This ensures the posterior probabilities across all classes sum to 1. In the medical example, P(positive) = P(positive|disease)*P(disease) + P(positive|no disease)*P(no disease). In Naive Bayes classification, we usually skip computing P(X) because it is the same for all classes — we just compare unnormalised posteriors to find the most probable class.
# Law of total probability: P(X) = sum_c P(X|c) * P(c)
classes = ['spam', 'ham', 'newsletter']
priors = [0.3, 0.5, 0.2] # Must sum to 1.0
P_word_given_class = [0.5, 0.1, 0.3] # Likelihood of 'free' in each class
# Total probability of seeing the word 'free'
P_free = sum(lik * pri for lik, pri in zip(P_word_given_class, priors))
print(f'P(free) = {P_free}')
# Posteriors (normalised)
for cls, pri, lik in zip(classes, priors, P_word_given_class):
posterior = lik * pri / P_free
print(f'P({cls} | free) = {posterior:.3f}')From Formula to Algorithm
Bayes' theorem translates directly into a classification algorithm. Given a labelled training set: Step 1: estimate P(class) for each class from class frequencies. Step 2: for each feature-class combination, estimate P(feature | class). Step 3: for a new sample, compute P(class | features) ∝ P(features | class) * P(class) for each class. Step 4: predict the class with the highest posterior. The 'naive' approximation in Naive Bayes simplifies Step 2: assume P(features|class) = product of P(feature_i | class) for all features.
# Naive Bayes algorithm in pseudocode
# Training:
# For each class c:
# prior[c] = count(class==c) / total_samples
# For each feature f:
# likelihood[c][f] = P(feature_f | class==c) # from training data
# Prediction for new sample x:
# For each class c:
# log_prob[c] = log(prior[c])
# For each feature f:
# log_prob[c] += log(likelihood[c][f=x_f]) # naive: assume independence
# return argmax(log_prob) # class with highest log-posterior
print('Naive independence assumption: P(f1,f2,...|c) = P(f1|c)*P(f2|c)*...*P(fn|c)')Why Log Probabilities Avoid Underflow
When multiplying many small probabilities together, the result quickly underflows to 0 in floating-point arithmetic — even though mathematically the value is a tiny but non-zero number. For example, a document with 100 words, each with P(word|class)=0.01, gives 0.01^100 = 10^{-200}, which is below the smallest float64 value. Working in log space converts multiplication to addition: log(a*b) = log(a) + log(b). Since we compare log posteriors (not actual posteriors), the argmax decision is identical. Naive Bayes implementations always operate in log space for numerical stability.
import numpy as np
# Direct multiplication: underflows
probs = [0.1] * 100 # 100 features, each P=0.1
product = np.prod(probs)
print('Direct product:', product) # 0.0 -- underflow!
# Log space: numerically stable
log_sum = np.sum(np.log(probs))
print('Log sum:', log_sum) # -100 * log(10) -- valid
# Compare two classes
log_prob_A = np.sum(np.log([0.1] * 100))
log_prob_B = np.sum(np.log([0.2] * 100))
print('Class A log-prob:', log_prob_A)
print('Class B log-prob:', log_prob_B)
print('Predicted:', 'A' if log_prob_A > log_prob_B else 'B')Bayes Optimal Classifier: The Gold Standard
The Bayes optimal classifier is the theoretically best possible classifier for a given data distribution. It predicts the class with the highest true posterior probability P(class|features). No other classifier can achieve a lower expected error rate on that distribution. In practice, we cannot use the Bayes optimal classifier because we do not know the true distributions — we can only estimate them from finite data. Naive Bayes is an approximation of the Bayes optimal classifier under the independence assumption. When that assumption holds, Naive Bayes is the optimal classifier.
# The Bayes error rate is the irreducible error
# Even a perfect model cannot beat it on a given distribution
# Example: predicting coin flip from noisy signal
import numpy as np
np.random.seed(42)
# True label: 50/50 coin
y_true = np.random.choice([0, 1], size=1000)
# Signal: 70% correlated with true label
signal = np.where(np.random.rand(1000) < 0.7, y_true, 1-y_true)
# Optimal prediction: just use the signal
acc = (signal == y_true).mean()
print(f'Optimal classifier accuracy: {acc:.3f}')
print(f'Bayes error rate (irreducible): {1-0.70:.3f} = 30%')Real-World Applications of Bayes' Theorem
Bayes' theorem is not just a classroom formula — it powers many real systems. Email spam filters compute P(spam | words) from word frequencies in known spam and ham. Medical diagnosis systems update disease probabilities as test results arrive. Search engines use Bayesian ranking to combine query relevance and document popularity. Autonomous vehicles use Bayesian filtering (Kalman filter, particle filter) to track their position given noisy sensor readings. A/B testing in industry increasingly uses Bayesian frameworks that output probability-of-being-best rather than p-values.
# Real-world Bayes: A/B test - which version is better?
# After 100 conversions from 1000 visitors for version A
# and 120 conversions from 1000 visitors for version B:
from scipy import stats
# Beta distribution as posterior for conversion rate
alpha_A, beta_A = 100 + 1, 900 + 1 # Beta(successes+1, failures+1)
alpha_B, beta_B = 120 + 1, 880 + 1
# Monte Carlo: P(B is better than A)
samples_A = stats.beta(alpha_A, beta_A).rvs(100000)
samples_B = stats.beta(alpha_B, beta_B).rvs(100000)
P_B_better = (samples_B > samples_A).mean()
print(f'P(B is better than A): {P_B_better:.3f}')The Independence Assumption: When Is It Valid?
The 'naive' part of Naive Bayes assumes features are conditionally independent given the class. For text data, this means assuming each word's presence is independent of other words (given the class). In reality, words like 'credit' and 'card' appear together — they are correlated. However, empirical studies show Naive Bayes works well despite violated independence because: (1) ranking for classification only needs the correct ordering of posteriors, not accurate probabilities; (2) with many features, independent signals combine well; (3) the simplicity prevents overfitting on small datasets.
# When does naive independence work?
# Even with correlated features, naive bayes often ranks classes correctly
import numpy as np
# Two correlated features that both indicate spam
words = {'buy': 0, 'now': 1, 'click': 2, 'here': 3}
# Spam email often has all four; ham rarely does
# Independence assumption ignores that 'buy' and 'now' co-occur
# But the combined signal is still strong for detecting spam
P_word_spam = [0.7, 0.6, 0.5, 0.6]
P_word_ham = [0.1, 0.1, 0.05, 0.1]
# For email with all 4 words
log_spam = np.sum(np.log(P_word_spam))
log_ham = np.sum(np.log(P_word_ham))
print('Log P(features|spam):', log_spam.round(2))
print('Log P(features|ham): ', log_ham.round(2))
print('Predicted: SPAM' if log_spam > log_ham else 'HAM')Bayes Factor: Comparing Hypotheses
The Bayes factor is the ratio of likelihoods for two competing hypotheses: BF = P(evidence | H1) / P(evidence | H0). Unlike p-values, which only tell you whether to reject the null hypothesis, the Bayes factor quantifies how much the evidence favours one hypothesis over another. BF > 10 is strong evidence for H1; BF < 0.1 is strong evidence for H0; values between 1/3 and 3 are inconclusive. Bayes factors are increasingly used in science and industry for A/B testing and hypothesis evaluation because they naturally incorporate prior information and give interpretable evidence strength.
# Bayes Factor example: comparing two models
# H0: coin is fair (p=0.5)
# H1: coin is biased (p=0.7)
# Observed: 8 heads in 10 flips
from scipy.stats import binom
observed_heads = 8
n_flips = 10
# Likelihood under each hypothesis
L_H0 = binom.pmf(observed_heads, n_flips, p=0.5)
L_H1 = binom.pmf(observed_heads, n_flips, p=0.7)
BF = L_H1 / L_H0
print(f'P(8 heads | fair coin) = {L_H0:.4f}')
print(f'P(8 heads | p=0.7) = {L_H1:.4f}')
print(f'Bayes Factor (H1/H0) = {BF:.2f}')
print('BF > 3: moderate evidence for biased coin')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: Bayes' theorem updates prior beliefs with evidence to produce posterior beliefs, the three components (prior, likelihood, posterior) and how they interact, and why low priors can dominate even high-accuracy tests (as in the medical diagnosis example). Next up we explore Bag of Words with CountVectorizer and TfidfVectorizer to convert text into numbers for Naive Bayes classification.
Frequently asked questions
Is the “Bayes' Theorem in Plain Language” lesson free?
Yes — the full text of “Bayes' Theorem in Plain Language” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Bayes' Theorem in Plain Language”?
Learners will work through a concrete medical-test example to build intuition for prior, likelihood, and posterior probabilities without heavy math. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Bayes' Theorem in Plain Language” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.