统计显著性与避免假阳性
学习可信实验背后的统计学知识，包括 p 值、样本量、统计效力，以及增长测试中导致虚假胜利的常见陷阱。
统计显著性与避免假阳性 是 CoddyKit 上的免费 ASO & App Growth 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 ASO & App Growth 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 ASO & App Growth 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
When Is a Win a Real Win?
Growth teams run constant experiments—but acting on false positives wastes effort and misleads strategy. Understanding statistical significance separates real wins from noise.
This lesson gives you the statistical literacy to trust your results.
Variation Is Everywhere
Even identical groups produce different numbers by chance. The core question of A/B testing is: is the observed difference real, or just random variation?
Statistics answers this rigorously.
Understanding the p-value
A p-value is the probability of seeing a result at least this extreme if there were truly no difference. A common threshold is p < 0.05.
p = 0.03 -> if there were no real effect,
only 3% chance of this result
-> usually deemed significant
p = 0.30 -> likely just noiseWhat Significance Does NOT Mean
p < 0.05 does not mean a 95% chance the hypothesis is true, nor that the effect is large. It only controls how often you wrongly cry "win" when nothing happened.
Statistical significance is not the same as business significance.
Sample Size Matters
Small samples produce wild swings and unreliable p-values. You need enough users per variant to detect the effect you care about.
Small sample: B looks +30%... then reverts
Large sample: stable, trustworthy estimate
Decide required sample size BEFORE launching.Statistical Power
Power is the chance of detecting a real effect if one exists (commonly targeted at 80%). Underpowered tests miss true winners—a false negative.
Power depends on sample size and effect size.
Minimum Detectable Effect
Before testing, set the minimum detectable effect (MDE)—the smallest improvement worth detecting. Smaller MDEs require larger samples.
Want to detect a 2% lift? -> need big sample
Only care about >10% lift? -> smaller sample okThe Peeking Problem
Checking results repeatedly and stopping the moment p < 0.05 dramatically inflates false positives. This peeking is one of the most common growth-testing mistakes.
Commit to a sample size and wait.
Multiple Comparisons
Test 20 variants at p < 0.05 and roughly one will look significant by pure chance. Testing many things at once inflates false positives—correct for it or pre-register your main metric.
Confidence Intervals Tell More
A confidence interval shows the plausible range of the true effect, not just a yes/no verdict. A wide interval crossing zero means you cannot be confident of direction—even with a tempting point estimate.
Building Trustworthy Experiments
A reliable test: define one primary metric, set MDE and sample size up front, run to completion without peeking, then judge on significance and practical impact.
Quick Check
Test your understanding of experiment statistics.
Recap: Significance & False Positives
You learned to run trustworthy experiments:
- A p-value measures the chance of a result under no real effect.
- Significance is not the same as a large or guaranteed-true effect.
- Set sample size, power, and MDE before launching.
- Avoid peeking and correct for multiple comparisons.
常见问题解答
「统计显著性与避免假阳性」课时是免费的吗？
是的 — 「统计显著性与避免假阳性」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 ASO & App Growth 课程的其余内容，请升级到 CoddyKit PRO。 ASO & App Growth 课程共包含 4 节课。
「统计显著性与避免假阳性」这节课中我会学到什么？
学习可信实验背后的统计学知识，包括 p 值、样本量、统计效力，以及增长测试中导致虚假胜利的常见陷阱。 你通过在浏览器中直接运行的动手代码来练习 ASO & App Growth，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 ASO & App Growth 需要有经验吗？
无需任何先前经验。CoddyKit 上的 ASO & App Growth 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「统计显著性与避免假阳性」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 ASO & App Growth 课中编写并运行代码吗？
能。每节 ASO & App Growth 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。
此课程中的所有课时
- 制定增长假设与框架
- 跨增长渠道开展 A/B 测试
- 迭代并扩展成功的实验
- 统计显著性与避免假阳性