0Pricing
R Academy · Lesson

Introduction to Bayesian Thinking

Understand prior, likelihood, and posterior in the Bayesian framework.

Introduction to Bayesian Thinking is a free R Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Frequentist vs Bayesian

In frequentist statistics, probability is the long-run frequency of an event. In Bayesian statistics, probability represents a degree of belief. The key difference: frequentists treat parameters as fixed (unknown constants); Bayesians treat parameters as random variables with probability distributions.

# Frequentist: parameter theta is fixed, data is random
# Bayesian: data is fixed (observed), theta has a distribution

# Example: estimating coin bias p
# Frequentist: MLE -> p_hat = heads / total
heads <- 7; total <- 10
p_mle <- heads / total
cat('MLE estimate:', p_mle, '\n')

# Bayesian: update prior belief with observed data
# Prior: Beta(2, 2) -> slightly informative, centered at 0.5
# Posterior: Beta(2 + heads, 2 + (total - heads)) = Beta(9, 5)
alpha_post <- 2 + heads
beta_post  <- 2 + (total - heads)
p_bayes <- alpha_post / (alpha_post + beta_post)  # posterior mean
cat('Bayesian posterior mean:', round(p_bayes, 3), '\n')

Bayes' Theorem

Bayes' theorem relates the prior belief about a parameter, the likelihood of the data given that parameter, and the posterior belief after observing the data:

P(θ|data) = P(data|θ) × P(θ) / P(data)

The denominator P(data) is a normalising constant — it makes the posterior a proper probability distribution.

# Bayes' theorem components
# P(theta | data)   = posterior  (what we want)
# P(data  | theta)  = likelihood (how well theta explains data)
# P(theta)          = prior      (what we believed before data)
# P(data)           = evidence   (normalising constant)

# Medical test example
P_disease   <- 0.01   # prior: 1% prevalence
P_pos_given_disease  <- 0.99  # sensitivity
P_pos_given_no_disease <- 0.05  # false positive rate

P_pos <- P_pos_given_disease * P_disease +
         P_pos_given_no_disease * (1 - P_disease)

P_disease_given_pos <- (P_pos_given_disease * P_disease) / P_pos

cat('P(positive test):', round(P_pos, 4), '\n')
cat('P(disease | positive test):', round(P_disease_given_pos, 4), '\n')
cat('Only', round(P_disease_given_pos * 100, 1), '% chance despite positive test!\n')

Prior Distributions

The prior encodes your belief about a parameter before seeing data. Priors can be uninformative (flat, expressing minimal knowledge) or informative (strongly peaked, expressing domain expertise). Common priors: Beta(1,1) = uniform, Normal(0, 10) = weakly informative.

# Visualise different Beta priors for a probability parameter
theta <- seq(0, 1, length.out = 200)

# Uniform (no prior knowledge)
prior_uniform <- dbeta(theta, 1, 1)

# Informative: believe p ~ 0.3
prior_informative <- dbeta(theta, 3, 7)

# Strong: believe p ~ 0.5
prior_strong <- dbeta(theta, 20, 20)

plot(theta, prior_uniform, type = 'l', col = 'gray',
     ylim = c(0, 8), xlab = 'theta', ylab = 'Density',
     main = 'Different Prior Beliefs')
lines(theta, prior_informative, col = 'blue')
lines(theta, prior_strong, col = 'red')
legend('topright', c('Uniform Beta(1,1)', 'Informative Beta(3,7)', 'Strong Beta(20,20)'),
       col = c('gray', 'blue', 'red'), lty = 1)

Likelihood Functions

The likelihood P(data|θ) measures how probable the observed data is for a given parameter value. For coin flips, the likelihood is Binomial. For continuous data, it's often Gaussian. We evaluate the likelihood at many θ values to find which best explains the data.

# Likelihood for a coin flip experiment
# Data: 7 heads in 10 flips
heads <- 7; n <- 10

# Evaluate likelihood at many theta values
theta <- seq(0.01, 0.99, length.out = 200)
likelihood <- dbinom(heads, n, theta)

# Maximum likelihood
mle <- theta[which.max(likelihood)]
cat('MLE (max likelihood theta):', mle, '\n')

# Plot the likelihood function
plot(theta, likelihood, type = 'l', col = 'steelblue',
     xlab = 'theta (coin bias)', ylab = 'Likelihood P(7H|theta)',
     main = '7 Heads in 10 Flips: Likelihood')
abline(v = mle, lty = 2, col = 'red')
legend('topleft', paste('MLE =', mle), lty = 2, col = 'red')

Posterior = Prior × Likelihood

The posterior is proportional to the prior times the likelihood. For the Beta-Binomial model this is analytically tractable: if the prior is Beta(α, β) and you observe h heads in n flips, the posterior is Beta(α + h, β + n − h).

# Beta-Binomial conjugate model
heads <- 7; n <- 10
alpha_prior <- 2; beta_prior <- 2  # prior: Beta(2,2)

# Update: posterior = Beta(alpha + heads, beta + tails)
alpha_post <- alpha_prior + heads
beta_post  <- beta_prior  + (n - heads)

theta <- seq(0.01, 0.99, length.out = 300)

prior     <- dbeta(theta, alpha_prior, beta_prior)
likelihood <- dbinom(heads, n, theta)
likelihood <- likelihood / max(likelihood)  # normalise for plotting
posterior  <- dbeta(theta, alpha_post, beta_post)

plot(theta, posterior,  type = 'l', col = 'red',  lwd = 2,
     xlab = 'theta', ylab = 'Density', main = 'Prior vs Posterior')
lines(theta, prior,      col = 'blue', lwd = 2)
lines(theta, likelihood, col = 'gray', lwd = 2, lty = 2)
legend('topleft', c(paste0('Prior Beta(', alpha_prior, ',', beta_prior, ')'),
                    'Likelihood (scaled)',
                    paste0('Posterior Beta(', alpha_post, ',', beta_post, ')')),
       col = c('blue', 'gray', 'red'), lty = c(1,2,1), lwd = 2)

Conjugate Priors

A conjugate prior is one where the posterior belongs to the same family as the prior. This makes inference analytically tractable. Common conjugate pairs: Beta-Binomial (proportions), Normal-Normal (means with known variance), Gamma-Poisson (rates).

# Conjugate prior table (analytical results)
conjugates <- data.frame(
  Likelihood    = c('Binomial',    'Poisson',    'Normal (known sigma)',
                    'Exponential', 'Multinomial'),
  Prior         = c('Beta',        'Gamma',      'Normal',
                    'Gamma',       'Dirichlet'),
  Posterior     = c('Beta',        'Gamma',      'Normal',
                    'Gamma',       'Dirichlet'),
  Update_Rule   = c('(a+h, b+t)',  '(a+x, b+n)', '(mu_n, sigma_n)',
                    '(a+n, b+sum)', '(a+counts)')
)
print(conjugates, row.names = FALSE)

# Beta-Binomial update
cat('\nBeta(2,3) + 7 heads, 3 tails -> Beta(',
    2+7, ',', 3+3, ')\n')

Credible Intervals

A credible interval (CI) is the Bayesian analog of a confidence interval: a 95% CI means there is a 95% posterior probability that θ lies in the interval. This is the intuitive interpretation most people incorrectly assign to frequentist confidence intervals.

# 95% credible interval for Beta posterior
alpha_post <- 9; beta_post <- 5  # posterior from earlier

# Credible interval via quantile function
ci_lower <- qbeta(0.025, alpha_post, beta_post)
ci_upper <- qbeta(0.975, alpha_post, beta_post)
posterior_mean <- alpha_post / (alpha_post + beta_post)

cat('Posterior mean: ', round(posterior_mean, 3), '\n')
cat('95% Credible Interval: [',
    round(ci_lower, 3), ',',
    round(ci_upper, 3), ']\n')
cat('Interpretation: 95% probability theta is in this interval\n')

# Visualise
theta <- seq(0, 1, length.out = 300)
plot(theta, dbeta(theta, alpha_post, beta_post), type = 'l', col = 'red', lwd = 2,
     main = '95% Credible Interval', xlab = 'theta', ylab = 'Density')
abline(v = c(ci_lower, ci_upper), lty = 2, col = 'blue')

Bayesian Updating: Sequential Learning

Bayesian updating is sequential: today's posterior becomes tomorrow's prior. This makes Bayesian inference naturally incremental — you don't need to re-fit from scratch when new data arrives, just update the existing posterior.

# Sequential Bayesian updating for a coin
# Start with uninformative prior Beta(1,1)
flips <- c(1, 0, 1, 1, 0, 1, 1, 1, 0, 1)  # 1=H, 0=T

alpha <- 1; beta_p <- 1  # prior
cat('Prior: Beta(', alpha, ',', beta_p, ') mean =', round(alpha/(alpha+beta_p), 3), '\n')

for (i in seq_along(flips)) {
  if (flips[i] == 1) alpha <- alpha + 1 else beta_p <- beta_p + 1
  mean_post <- alpha / (alpha + beta_p)
  cat('After flip', i, '(', flips[i], '): Beta(',
      alpha, ',', beta_p, ') mean =', round(mean_post, 3), '\n')
}

MAP Estimation

The Maximum A Posteriori (MAP) estimate is the mode of the posterior distribution. For a Beta(α, β) posterior, MAP = (α−1)/(α+β−2). MAP balances the prior's pull with the likelihood's pull, shrinking estimates toward the prior for small samples.

# Compare MLE vs MAP for small sample
heads <- 3; n <- 5
alpha_p <- 5; beta_p <- 5  # informative prior: believe p ~ 0.5

# MLE: ignores prior
mle <- heads / n

# MAP: mode of Beta posterior
alpha_post <- alpha_p + heads
beta_post  <- beta_p  + (n - heads)
map <- (alpha_post - 1) / (alpha_post + beta_post - 2)

# Posterior mean (alternative point estimate)
post_mean <- alpha_post / (alpha_post + beta_post)

cat('Data: 3 heads in 5 flips\n')
cat('MLE:           ', round(mle, 3), '(ignores prior)\n')
cat('MAP:           ', round(map, 3), '(mode of posterior)\n')
cat('Posterior mean:', round(post_mean, 3), '(mean of posterior)\n')
cat('Note: MAP and mean shrink toward prior (0.5) for small n\n')

When to Use Bayesian Methods

Bayesian methods shine when: (1) you have informative prior knowledge, (2) sample sizes are small, (3) you need full uncertainty quantification, (4) you want to make probability statements about parameters, or (5) you are doing sequential analysis where priors carry over from previous experiments.

# Comparison: when Bayesian vs frequentist is preferred
comparison <- data.frame(
  Scenario = c(
    'Small sample (n < 30)',
    'Prior domain knowledge',
    'Probability about parameter',
    'Sequential updating',
    'Large sample, no prior',
    'Regulatory/simple inference'
  ),
  Preferred = c(
    'Bayesian', 'Bayesian', 'Bayesian',
    'Bayesian', 'Either',   'Frequentist'
  )
)
print(comparison, row.names = FALSE)

# Example: medical device testing with historical data
alpha_historical <- 15  # prior based on 20 historical tests
beta_historical  <- 5
cat('\nHistorical prior: Beta(', alpha_historical, ',', beta_historical, ')\n')
cat('Prior mean:', round(alpha_historical/(alpha_historical+beta_historical), 3), '\n')

Bayesian Inference in Practice

For simple conjugate models, inference is analytical (as shown above). For complex models (hierarchical, non-conjugate), we use Markov Chain Monte Carlo (MCMC) sampling via Stan (RStan), JAGS, or BUGS to approximate the posterior numerically.

# Analytical vs MCMC approaches
approaches <- data.frame(
  Method        = c('Conjugate (exact)', 'Grid approximation',
                    'Laplace approx.', 'MCMC (Stan/JAGS)',
                    'Variational Bayes'),
  When          = c('Conjugate prior+likelihood', 'Low-dim, discrete',
                    'Unimodal posterior', 'General complex models',
                    'Large scale, approximate'),
  Speed         = c('Instant', 'Fast', 'Fast', 'Slow', 'Moderate'),
  Exactness     = c('Exact', 'Exact on grid', 'Approximate',
                    'Asymptotically exact', 'Approximate')
)
print(approaches, row.names = FALSE)

Quick Check

You observe 7 heads in 10 coin flips. Your prior is Beta(2, 2). What is the correct posterior distribution?

Recap: Bayesian Thinking

Key takeaways:

  • Bayes' theorem: P(θ|data) ∝ P(data|θ) × P(θ)
  • Prior encodes belief before data; likelihood encodes data support; posterior combines both
  • Conjugate priors give analytical posteriors (Beta-Binomial, Normal-Normal, Gamma-Poisson)
  • Beta-Binomial update: Beta(α, β) + (h heads, t tails) → Beta(α+h, β+t)
  • Credible intervals have the natural probability interpretation that CIs do not
  • Bayesian updating is sequential — today's posterior is tomorrow's prior
  • For complex models, use MCMC sampling (Stan, JAGS) to approximate posteriors
# Full Bayesian inference cycle for a proportion
alpha0 <- 2; beta0 <- 2  # prior
heads  <- 12; total <- 20  # observed data

alpha_post <- alpha0 + heads
beta_post  <- beta0  + (total - heads)

post_mean  <- alpha_post / (alpha_post + beta_post)
ci         <- qbeta(c(0.025, 0.975), alpha_post, beta_post)

cat('Prior: Beta(', alpha0, ',', beta0, ') mean =', round(alpha0/(alpha0+beta0), 2), '\n')
cat('Data:', heads, 'heads in', total, 'flips\n')
cat('Posterior: Beta(', alpha_post, ',', beta_post, ')\n')
cat('Posterior mean:', round(post_mean, 3), '\n')
cat('95% CI: [', round(ci[1],3), ',', round(ci[2],3), ']\n')

Frequently asked questions

Is the “Introduction to Bayesian Thinking” lesson free?

Yes — the full text of “Introduction to Bayesian Thinking” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.

What will I learn in “Introduction to Bayesian Thinking”?

Understand prior, likelihood, and posterior in the Bayesian framework. You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start R Academy?

No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Introduction to Bayesian Thinking” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this R Academy lesson?

Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Introduction to Bayesian Thinking
  2. Writing Stan Models in R
  3. MCMC Sampling and Diagnostics
  4. Posterior Predictive Checks
← Back to R Academy