T-Tests for Comparing Means

A t-test is a hypothesis test that determines whether a difference in means between groups is statistically significant or likely due to random chance. It compares the observed difference to the variability in the data, producing a t-statistic and a p-value. If p ≤ 0.05 (the conventional threshold), we reject the null hypothesis that the means are equal. There are three common types: one-sample, independent two-sample, and paired t-test, each for different experimental designs.

The one-sample t-test tests whether the mean of a sample differs significantly from a known or hypothesised population mean. For example: 'Is our average delivery time significantly different from the industry standard of 3 days?' Use scipy.stats.ttest_1samp(data, popmean). The null hypothesis is H₀: mean = popmean. A small p-value leads you to reject H₀ and conclude the sample mean differs from the target.

The independent two-sample t-test compares the means of two independent groups. For example: 'Does the A/B test variant produce higher average revenue than the control?' Use scipy.stats.ttest_ind(group_a, group_b). The equal_var parameter matters: set equal_var=False (Welch's t-test) when the two groups may have different variances, which is the safe default for most real data.

By default, t-tests are two-tailed: they test whether means differ in either direction (greater OR less than). If you have a directional hypothesis ('the variant increases revenue'), use a one-tailed test with the alternative parameter: 'greater' or 'less'. A one-tailed test has more power for the specific direction but provides no evidence about the other direction. Decide on the direction before looking at the data to avoid HARKing (Hypothesising After Results are Known).

The paired t-test is used when each observation in group A has a natural partner in group B — for example, measurements before and after a treatment on the same subjects, or the same stores in two different months. Pairing controls for between-subject variability, making the test more powerful than an independent t-test for the same data. Use scipy.stats.ttest_rel(before, after). The order of elements must be matched: before[i] and after[i] must come from the same subject.

The t-statistic measures how many standard errors the observed difference is from zero. A t-statistic of 2 means the observed difference is 2 standard errors above what would be expected under the null. For a two-tailed test at α=0.05 with a large sample, the critical value is approximately ±1.96 — so |t| > 1.96 gives p < 0.05. The p-value converts the t-statistic to a probability, making it easier to interpret without consulting t-distribution tables.

A p-value alone does not tell you the magnitude of the effect. Always compute a confidence interval for the mean difference. A 95% CI that includes zero is consistent with p > 0.05 (not significant); one that excludes zero means the difference is significant. The CI also tells you whether the effect size is practically meaningful — a statistically significant difference of $0.01 in revenue may be irrelevant, while a $50 difference with p=0.06 may still matter for the business.

Statistical significance depends on sample size — with large enough samples, even trivial differences become significant. Cohen's d measures practical significance (effect size): the mean difference divided by the pooled standard deviation. Guidelines: d < 0.2 is negligible, 0.2–0.5 is small, 0.5–0.8 is medium, > 0.8 is large. Always report effect size alongside p-values — a significant p-value with d = 0.05 means the difference is real but probably not meaningful in practice.

The independent t-test assumes: (1) Independence — observations within each group are independent of each other; (2) Normality — data in each group is approximately normal (robust for n > 30 via CLT); (3) for Student's t-test (equal_var=True), equal variances. Welch's t-test (equal_var=False) relaxes assumption 3 and is preferred. When normality is seriously violated with small samples (< 30), use the Mann-Whitney U test instead.

In practice, the data you want to test lives in a Pandas DataFrame column. You can pass a Pandas Series directly to scipy.stats functions — they work seamlessly with Series as well as NumPy arrays. A common workflow: filter the DataFrame to get each group's Series, run the t-test, and store results in a summary DataFrame. This pattern scales to testing many pairs of columns or groups in a loop.

Running many t-tests simultaneously inflates the chance of a false positive. If you run 20 t-tests at α=0.05, you expect 1 false positive by chance. This is the multiple comparisons problem. Apply the Bonferroni correction: divide α by the number of tests (p_threshold = 0.05 / n_tests). For more power with less conservatism, use the Benjamini-Hochberg False Discovery Rate (FDR) correction available in statsmodels. Always correct for multiple comparisons in A/B testing with many metrics.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: ttest_1samp tests a sample mean against a reference value, ttest_ind (Welch's with equal_var=False) compares two independent groups, and ttest_rel handles paired measurements. Always report Cohen's d alongside the p-value to distinguish statistical from practical significance. Next up we test relationships between categorical variables with the chi-squared test.