Hypothesis Testing
Null hypothesis, p-value, t-test, chi-square test with scipy.stats.
Hypothesis Testing is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What is a Hypothesis Test?
A hypothesis test uses data to decide whether an observed effect is real or just random chance. It is how data scientists justify claims like "the new design increased clicks".
Null vs Alternative
The null hypothesis H0 says there is NO effect (the default, skeptical position). The alternative H1 says there IS an effect. We try to gather evidence against H0.
The p-value
The p-value is the probability of seeing data this extreme IF H0 were true. A small p-value means the data is surprising under H0, so we doubt H0.
Significance Level
We compare p to a threshold alpha (commonly 0.05). If p < alpha we reject H0; otherwise we fail to reject it. alpha is the acceptable false-positive rate.
What p Does NOT Mean
The p-value is NOT the probability that H0 is true, and NOT the size of the effect. A tiny p with a huge sample can reflect a trivially small difference. Always report effect size too.
One-Sample t-test
ttest_1samp checks whether a sample mean differs from a known value, for example whether average response time differs from a target of 200 ms.
from scipy import stats
import numpy as np
times = np.array([198, 205, 210, 195, 220, 208])
t, p = stats.ttest_1samp(times, popmean=200)
print(t, p)Two-Sample t-test
ttest_ind compares the means of two INDEPENDENT groups, the standard A/B test analysis.
control = np.array([20, 22, 19, 24, 21])
treat = np.array([25, 27, 24, 29, 26])
t, p = stats.ttest_ind(control, treat)
print(t, p) # small p -> groups differInterpreting the Result
Read the p-value against alpha and state a conclusion in plain language. Significance is a decision, not proof of certainty.
alpha = 0.05
if p < alpha:
print("Reject H0: the groups differ significantly")
else:
print("Fail to reject H0: no significant difference")Chi-Square Test of Independence
chi2_contingency tests whether two CATEGORICAL variables are associated, using a contingency table of counts.
table = np.array([[30, 10], [20, 40]]) # rows: group, cols: outcome
chi2, p, dof, expected = stats.chi2_contingency(table)
print(chi2, p)Type I and Type II Errors
A Type I error (false positive) rejects a true H0; its rate is alpha. A Type II error (false negative) fails to reject a false H0; its rate is beta. Power = 1 - beta is the chance of detecting a real effect.
Trade-offs
Lowering alpha reduces false positives but raises the false-negative rate (lower power). Larger samples reduce both error types, which is why sample size planning matters before an experiment.
Quick Check
Test your hypothesis-testing knowledge.
Recap
Hypothesis-testing essentials:
- H0 (no effect) vs H1 (effect); gather evidence against H0
- p-value = P(data this extreme | H0); reject if p < alpha
- p is NOT P(H0 true) and NOT effect size
ttest_1samp,ttest_ind,chi2_contingency- Type I (false positive, rate alpha) vs Type II (false negative, rate beta); power = 1-beta
Frequently asked questions
Is the “Hypothesis Testing” lesson free?
Yes — the full text of “Hypothesis Testing” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Hypothesis Testing”?
Null hypothesis, p-value, t-test, chi-square test with scipy.stats. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Hypothesis Testing” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.