Statistical Significance

Conversion rates jump around by chance. If variant B beats A by 2% on a small sample, that gap might be pure noise.

Statistical significance tells you how likely the observed difference is real rather than random luck. Without it, you risk shipping changes that do nothing.

Every measurement has variation. Flip a fair coin ten times and you rarely get exactly five heads.

The same applies to conversions. A short test can show a fake winner simply because one variant got a lucky streak. Significance separates genuine signal from this noise.

The p-value estimates the probability of seeing your result, or a more extreme one, if there were truly no difference between variants.

A small p-value means the result would be unlikely by chance alone. Marketers commonly treat p below 0.05 as the threshold for declaring significance.

Most A/B tools report a confidence level instead of a raw p-value. A 95% confidence level corresponds to p below 0.05.

It means: if nothing really changed, you would see a difference this large only about 5% of the time. Higher confidence demands more data.

Power is the chance your test detects a real effect when one exists. Low power means real winners slip by unnoticed.

The standard target is 80% power. Underpowered tests waste traffic and produce inconclusive results that tempt you to call them too early.

The minimum detectable effect, or MDE, is the smallest improvement you want the test to reliably catch.

Detecting a tiny 1% lift needs far more traffic than catching a 10% lift. Set a realistic MDE before launch; it directly determines your required sample size.

Before running, plug your baseline rate, target MDE, confidence, and power into a sample-size calculator. It tells you how many visitors each variant needs.

Commit to that number. Ending early because B "looks like it's winning" is the most common way teams fool themselves.

Checking results repeatedly and stopping the moment significance appears inflates your false-positive rate dramatically.

Every peek is another chance for random noise to cross the line. Decide your sample size and duration in advance, then wait. Do not stop on the first green result.

Behavior differs by weekday, weekend, and payday. A test that runs Monday to Wednesday misses weekend shoppers entirely.

Run for at least one or two full weeks so each variant sees a representative mix of traffic. Sample size and time both gate a valid result.

A result can be statistically significant yet too small to matter for the business. With huge traffic, even a 0.1% lift can hit 95% confidence.

Always pair significance with the effect size and its practical value. Ask: is this lift big enough to justify the change?

The confidence interval shows the plausible range of the true effect. A point estimate of +8% with an interval of -1% to +17% is shaky.

Prefer results where the entire interval stays positive. A wide interval that crosses zero means you cannot be sure the change helps at all.

Test your understanding of significance.

Significance separates real effects from random noise. Aim for 95% confidence and 80% power, set an MDE, and calculate sample size before launching.

Avoid peeking, run full business cycles, and judge results by effect size and confidence interval, not significance alone.