ANOVA and Post-Hoc Tests
Compare means across three or more groups with one-way ANOVA and run Tukey HSD to identify which pairs differ.
Why Not Multiple T-Tests?
When comparing means across three or more groups, running multiple t-tests inflates the Type I error rate (false positive rate). Testing groups A vs B, A vs C, and B vs C at α=0.05 each gives an overall false positive chance of about 14%, not 5%. Analysis of Variance (ANOVA) solves this by testing all groups simultaneously in a single test, keeping the false positive rate at exactly α. One ANOVA replaces all pairwise t-tests for the omnibus question: 'Do any groups differ?'
One-Way ANOVA: The Big Picture
One-way ANOVA tests whether the means of three or more groups differ significantly. It decomposes total variance into between-group variance (explained by group membership) and within-group variance (random noise). The F-statistic is their ratio: F = (between-group variance) / (within-group variance). A large F means group differences are large relative to noise, yielding a small p-value. The null hypothesis is H₀: all group means are equal.
import numpy as np
from scipy import stats
np.random.seed(0)
# Three product variants with different conversion rates
group_a = np.random.normal(10.0, 2.0, 50) # baseline
group_b = np.random.normal(11.5, 2.0, 50) # slightly better
group_c = np.random.normal(13.0, 2.0, 50) # clearly better
f_stat, p = stats.f_oneway(group_a, group_b, group_c)
print(f'F-statistic: {f_stat:.3f}')
print(f'p-value: {p:.6f}')
print('At least one group differs?', 'Yes' if p < 0.05 else 'No')