How do you assess statistical significance in an A/B test?

Example Answer

I use the analysis plan agreed before the test: the hypothesis, metric, appropriate test, and significance threshold. For a valid fixed-horizon test, a p-value below that threshold is evidence against the null hypothesis. I also report the estimated effect and confidence interval, because statistical significance alone does not show that a change is worthwhile.

Checks before deciding

Confirm that the data and randomization unit match the test’s assumptions. For example, repeated events from the same user are not automatically independent observations. Check allocation and logging quality, and account for multiple comparisons when testing several outcomes or variants.

If the result is not significant, say that the test did not provide sufficient evidence under the chosen plan; do not claim it proved zero effect. Discuss the uncertainty and whether the test had enough sensitivity to detect a meaningful change. Guardrails and implementation cost also matter to the release decision.

Reference

NIST: statistical tests, Microsoft Research: post-experiment checks