ANOVA and Post-Hoc Tests
Compare means across three or more groups with one-way ANOVA and run Tukey HSD to identify which pairs differ.
ANOVA and Post-Hoc Tests is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Not Multiple T-Tests?
When comparing means across three or more groups, running multiple t-tests inflates the Type I error rate (false positive rate). Testing groups A vs B, A vs C, and B vs C at α=0.05 each gives an overall false positive chance of about 14%, not 5%. Analysis of Variance (ANOVA) solves this by testing all groups simultaneously in a single test, keeping the false positive rate at exactly α. One ANOVA replaces all pairwise t-tests for the omnibus question: 'Do any groups differ?'
One-Way ANOVA: The Big Picture
One-way ANOVA tests whether the means of three or more groups differ significantly. It decomposes total variance into between-group variance (explained by group membership) and within-group variance (random noise). The F-statistic is their ratio: F = (between-group variance) / (within-group variance). A large F means group differences are large relative to noise, yielding a small p-value. The null hypothesis is H₀: all group means are equal.
import numpy as np
from scipy import stats
np.random.seed(0)
# Three product variants with different conversion rates
group_a = np.random.normal(10.0, 2.0, 50) # baseline
group_b = np.random.normal(11.5, 2.0, 50) # slightly better
group_c = np.random.normal(13.0, 2.0, 50) # clearly better
f_stat, p = stats.f_oneway(group_a, group_b, group_c)
print(f'F-statistic: {f_stat:.3f}')
print(f'p-value: {p:.6f}')
print('At least one group differs?', 'Yes' if p < 0.05 else 'No')ANOVA with Pandas DataFrames
In practice, your data is in a long-format DataFrame where one column holds the group label and another holds the measurement. Extract each group as a Series and pass them to stats.f_oneway(). Alternatively, if you have a wide-format DataFrame (one column per group), pass each column directly. The function accepts any number of positional arguments, making it easy to extend to 4, 5, or more groups.
import pandas as pd
import numpy as np
from scipy import stats
np.random.seed(1)
df = pd.DataFrame({
'group': ['A']*40 + ['B']*40 + ['C']*40 + ['D']*40,
'score': np.concatenate([
np.random.normal(70, 10, 40),
np.random.normal(75, 10, 40),
np.random.normal(80, 10, 40),
np.random.normal(72, 10, 40)
])
})
groups = [group['score'].values
for _, group in df.groupby('group')]
f, p = stats.f_oneway(*groups)
print(f'F={f:.3f}, p={p:.4f}')
print(df.groupby('group')['score'].mean().round(2))ANOVA Assumptions
One-way ANOVA assumes: (1) Independence — observations are independent of each other; (2) Normality — residuals are approximately normal within each group (robust for large samples via CLT); (3) Homoscedasticity — equal variances across groups. Test the variance assumption with Levene's test (stats.levene(*groups)). If variances are unequal, use Welch's ANOVA instead, available in the pingouin library or computed via statsmodels.
import numpy as np
from scipy import stats
np.random.seed(0)
g1 = np.random.normal(10, 2, 40)
g2 = np.random.normal(12, 2, 40)
g3 = np.random.normal(11, 8, 40) # Much higher variance
# Test equal variances
stat_l, p_l = stats.levene(g1, g2, g3)
print(f'Levene\'s test: stat={stat_l:.3f}, p={p_l:.4f}')
if p_l < 0.05:
print('Warning: Unequal variances! Standard ANOVA may be unreliable.')
print('Consider Welch\'s ANOVA or Kruskal-Wallis.')Kruskal-Wallis: Non-Parametric ANOVA
When ANOVA assumptions are violated (non-normal distributions, small samples, unequal variances), the Kruskal-Wallis test is the non-parametric alternative. It tests whether the distributions of groups differ by converting data to ranks and comparing ranked sums. Use scipy.stats.kruskal(*groups). The null hypothesis is that all groups have the same distribution (equivalent to testing equal medians when distributions have the same shape). It is always valid but less powerful than ANOVA when assumptions hold.
import numpy as np
from scipy import stats
np.random.seed(0)
# Non-normal data: customer satisfaction scores (1-10 scale)
g1 = np.random.choice(range(1, 11), 50, p=[0.05]*10)
g2 = np.random.choice(range(1, 11), 50, p=[0.02, 0.03, 0.05, 0.08, 0.12, 0.15, 0.20, 0.15, 0.12, 0.08])
g3 = np.random.choice(range(1, 11), 50, p=[0.15, 0.15, 0.15, 0.15, 0.10, 0.10, 0.08, 0.05, 0.04, 0.03])
stat, p = stats.kruskal(g1, g2, g3)
print(f'Kruskal-Wallis: H={stat:.3f}, p={p:.4f}')
print('Groups differ?', 'Yes' if p < 0.05 else 'No')Post-Hoc Tests: Why You Need Them
A significant ANOVA tells you that at least one group mean differs, but not which pairs differ. Post-hoc tests perform all pairwise comparisons while correcting for multiple comparisons. The most popular is Tukey's Honest Significant Difference (Tukey HSD), which controls the family-wise error rate exactly at α. It is available in statsmodels.stats.multicomp.pairwise_tukeyhsd(). Run post-hoc tests only after a significant ANOVA — not as a fishing expedition on non-significant results.
import numpy as np
import pandas as pd
from scipy import stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd
np.random.seed(0)
data = np.concatenate([
np.random.normal(70, 10, 40),
np.random.normal(80, 10, 40),
np.random.normal(75, 10, 40)
])
groups = ['A']*40 + ['B']*40 + ['C']*40
# First confirm ANOVA is significant
f, p = stats.f_oneway(*[data[i*40:(i+1)*40] for i in range(3)])
print(f'ANOVA p={p:.4f}')
if p < 0.05:
print('Post-hoc Tukey HSD:')
print(pairwise_tukeyhsd(data, groups, alpha=0.05))Reading Tukey HSD Output
The Tukey HSD table has one row per pair of groups. The columns to focus on are: meandiff (mean difference between the pair), lower and upper (95% confidence interval for the difference), and reject (True if the pair differs significantly at α). A CI that does not include zero means the pair is significantly different. Groups whose CIs overlap are not significantly different from each other.
import numpy as np
from statsmodels.stats.multicomp import pairwise_tukeyhsd
np.random.seed(42)
data = np.concatenate([
np.random.normal(10, 2, 60), # Group A
np.random.normal(13, 2, 60), # Group B (clearly higher)
np.random.normal(10.5, 2, 60) # Group C (barely different from A)
])
groups = ['A']*60 + ['B']*60 + ['C']*60
result = pairwise_tukeyhsd(data, groups, alpha=0.05)
print(result)
# A-B: reject=True (B is higher)
# A-C: reject=False (barely different)
# B-C: reject=True (B is higher)Bonferroni and Benjamini-Hochberg Corrections
Alternative to Tukey HSD, you can apply a Bonferroni correction to pairwise t-tests: run all t-tests, then adjust the threshold to α/k where k is the number of comparisons. This is conservative — it may miss real differences. The Benjamini-Hochberg False Discovery Rate (FDR) correction is less conservative: it controls the expected proportion of false discoveries, allowing more true positives at the cost of slightly more false positives. Use statsmodels.stats.multitest.multipletests(p_values, method='fdr_bh').
import numpy as np
from scipy import stats
from statsmodels.stats.multitest import multipletests
np.random.seed(0)
groups = [np.random.normal(10 + i*1.5, 2, 40) for i in range(4)]
names = ['A', 'B', 'C', 'D']
# All pairwise t-tests
p_values = []
pairs = []
for i in range(len(groups)):
for j in range(i+1, len(groups)):
_, p = stats.ttest_ind(groups[i], groups[j])
p_values.append(p)
pairs.append(f'{names[i]}-{names[j]}')
# Benjamini-Hochberg correction
reject, adj_p, _, _ = multipletests(p_values, method='fdr_bh')
for pair, p, ap, r in zip(pairs, p_values, adj_p, reject):
print(f'{pair}: raw_p={p:.4f}, adj_p={ap:.4f}, significant={r}')Effect Size for ANOVA: Eta-Squared
For ANOVA, the analogous measure to Cohen's d is eta-squared (η²): the proportion of total variance explained by group membership. η² = SS_between / SS_total. Values: < 0.01 is small, 0.06 is medium, > 0.14 is large. Partial eta-squared (η²_p) is more common in published research: it divides only by the group variance plus the error variance, not the total. Always compute η² alongside the ANOVA p-value to show whether the effect is practically meaningful, not just statistically detectable.
import numpy as np
from scipy import stats
np.random.seed(0)
groups = [
np.random.normal(10, 2, 50),
np.random.normal(12, 2, 50),
np.random.normal(14, 2, 50)
]
all_data = np.concatenate(groups)
grand_mean = all_data.mean()
ss_between = sum(len(g) * (g.mean() - grand_mean)**2 for g in groups)
ss_total = sum((x - grand_mean)**2 for g in groups for x in g)
eta_sq = ss_between / ss_total
f, p = stats.f_oneway(*groups)
print(f'F={f:.3f}, p={p:.4f}')
print(f'Eta-squared: {eta_sq:.4f} ({eta_sq:.1%} of variance explained)')Two-Way ANOVA Overview
Two-way ANOVA extends one-way ANOVA to two categorical factors simultaneously. For example, testing whether exam scores differ by both teaching method and student experience level simultaneously. It also tests the interaction effect: does the effect of teaching method depend on experience level? Two-way ANOVA is implemented in statsmodels via ols('score ~ C(method) + C(level) + C(method):C(level)', data=df).fit() and anova_lm(model). This is a foundation for more advanced experimental design.
import pandas as pd
import numpy as np
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm
np.random.seed(0)
df = pd.DataFrame({
'method': ['trad']*60 + ['active']*60,
'level': ['beginner']*30 + ['advanced']*30 + ['beginner']*30 + ['advanced']*30,
'score': (np.random.normal(70, 8, 30).tolist() +
np.random.normal(80, 8, 30).tolist() +
np.random.normal(75, 8, 30).tolist() +
np.random.normal(88, 8, 30).tolist())
})
model = ols('score ~ C(method) + C(level) + C(method):C(level)', data=df).fit()
print(anova_lm(model, typ=2)[['F', 'PR(>F)']].round(4))Repeated Measures ANOVA
Repeated measures ANOVA is used when the same subjects are measured multiple times — for example, testing reaction time at time points 0, 30, and 60 minutes after a treatment. It is the multi-group generalisation of the paired t-test. By accounting for individual differences, it has more power than one-way ANOVA. In Python, use pingouin.rm_anova() or pg.pairwise_tests() for the post-hoc analysis. When sphericity is violated (tested with Mauchly's test), use Greenhouse-Geisser corrected p-values.
import pandas as pd
import numpy as np
# Example structure for repeated measures (pingouin library)
np.random.seed(0)
subjects = list(range(20)) * 3 # 20 subjects, 3 time points
timepoints = ['T0']*20 + ['T1']*20 + ['T2']*20
scores = np.concatenate([
np.random.normal(50, 10, 20), # baseline
np.random.normal(55, 10, 20), # improved
np.random.normal(60, 10, 20) # further improved
])
df_rm = pd.DataFrame({'subject': subjects, 'time': timepoints, 'score': scores})
print(df_rm.groupby('time')['score'].mean().round(2))
# Use pingouin.rm_anova(data=df_rm, dv='score', within='time', subject='subject')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: scipy.stats.f_oneway() tests whether any groups differ in mean while controlling the overall Type I error rate, post-hoc tests (Tukey HSD) identify which specific pairs differ after a significant ANOVA, and Kruskal-Wallis is the non-parametric fallback when normality or equal-variance assumptions are violated. Next up we begin the capstone project, integrating all skills into an end-to-end data pipeline.
Frequently asked questions
Is the “ANOVA and Post-Hoc Tests” lesson free?
Yes — the full text of “ANOVA and Post-Hoc Tests” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “ANOVA and Post-Hoc Tests”?
Compare means across three or more groups with one-way ANOVA and run Tukey HSD to identify which pairs differ. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “ANOVA and Post-Hoc Tests” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.