Statistical Significance
Sample size and confidence.
Why Significance Matters
Conversion rates jump around by chance. If variant B beats A by 2% on a small sample, that gap might be pure noise.
Statistical significance tells you how likely the observed difference is real rather than random luck. Without it, you risk shipping changes that do nothing.
Signal Versus Noise
Every measurement has variation. Flip a fair coin ten times and you rarely get exactly five heads.
The same applies to conversions. A short test can show a fake winner simply because one variant got a lucky streak. Significance separates genuine signal from this noise.
Day 1: B looks +14% better
Day 3: gap shrinks to +4%
Day 7: gap settles to +1% (noise)
Early leads often regress to the mean.All lessons in this course
- What to Test
- Building a Hypothesis
- Statistical Significance
- Reading Results