통계적 유의성과 거짓 양성 피하기

Growth teams run constant experiments—but acting on false positives wastes effort and misleads strategy. Understanding statistical significance separates real wins from noise.

This lesson gives you the statistical literacy to trust your results.

Even identical groups produce different numbers by chance. The core question of A/B testing is: is the observed difference real, or just random variation?

Statistics answers this rigorously.

A p-value is the probability of seeing a result at least this extreme if there were truly no difference. A common threshold is p < 0.05.

p < 0.05 does not mean a 95% chance the hypothesis is true, nor that the effect is large. It only controls how often you wrongly cry "win" when nothing happened.

Statistical significance is not the same as business significance.

Small samples produce wild swings and unreliable p-values. You need enough users per variant to detect the effect you care about.

Power is the chance of detecting a real effect if one exists (commonly targeted at 80%). Underpowered tests miss true winners—a false negative.

Power depends on sample size and effect size.

Before testing, set the minimum detectable effect (MDE)—the smallest improvement worth detecting. Smaller MDEs require larger samples.

Checking results repeatedly and stopping the moment p < 0.05 dramatically inflates false positives. This peeking is one of the most common growth-testing mistakes.

Commit to a sample size and wait.

Test 20 variants at p < 0.05 and roughly one will look significant by pure chance. Testing many things at once inflates false positives—correct for it or pre-register your main metric.

A confidence interval shows the plausible range of the true effect, not just a yes/no verdict. A wide interval crossing zero means you cannot be confident of direction—even with a tempting point estimate.

A reliable test: define one primary metric, set MDE and sample size up front, run to completion without peeking, then judge on significance and practical impact.

You learned to run trustworthy experiments:

• A p-value measures the chance of a result under no real effect.
• Significance is not the same as a large or guaranteed-true effect.
• Set sample size, power, and MDE before launching.
• Avoid peeking and correct for multiple comparisons.