統計的有意性を正しく読み取る

Early in a test, the challenger might look amazing purely by luck. Statistical significance is how you tell a real effect from random noise. 🎲

A p-value estimates the chance of seeing your result if the two models were truly equal. A small p-value means the gap is unlikely to be pure luck.

Teams often call a result significant when the p-value drops below 0.05. It is a convention, not a law, so treat it as a guide rather than gospel.

For a conversion test, a two-proportion z-test compares the two rates. SciPy can run it from your group counts.

A confidence interval shows the plausible range of the true lift. If that range still includes zero, you cannot yet claim a real difference.

Decide how many users you need before starting, based on the lift you hope to detect. Too few users and even a true win stays invisible.

Checking results over and over and stopping the moment it looks good is peeking. It massively inflates false positives, so resist the urge to call it early.

Commit to a fixed end date or sample size up front and let the test finish. Weekday and weekend users differ, so a full cycle avoids skew.

A result can be statistically significant yet tiny. Always ask if the effect size is large enough to justify shipping the new model at all.

Test twenty metrics and one will likely look significant by chance. Stick to your one primary metric, or correct for testing many at once.

The whole point of significance is to stop you fooling yourself. Pre-register your plan, wait for enough data, and report the result honestly. ✅

Let us catch the most common self-deception.

Use a p-value and confidence interval to separate signal from noise. Fix your sample size first, never peek, and weigh effect size before you ship. 🎯