How would you determine the sample size needed for an A/B test?

Official answer

Specify the primary outcome, assignment unit, comparison and analysis before calculating sample size. Choose a significance level, sidedness, target power, allocation ratio and planning effect. Estimate baseline variability using relevant historical data. A statistically detectable effect should be compared with the smallest effect worth acting on; these are related planning choices, not interchangeable definitions.

For an illustrative independent binary conversion outcome, suppose control conversion is 10%, the planning alternative is 11%, allocation is equal, the test is two-sided at 5%, and target power is 80%. The difference is one percentage point, or 10% relative to baseline.

A common large-sample approximation per group is

n ≈ [z(0.975)√(2p̄(1−p̄)) + z(0.80)√(p0(1−p0)+p1(1−p1))]² / (p1−p0)², with p̄=(p0+p1)/2.

Using p0=0.10 and p1=0.11 gives about 14,751 units per group, rounded up. This is a planning approximation without continuity correction, not a universal sample requirement. Check the selected method with software or simulation, particularly for sparse outcomes or complex estimators.

Adjust for clustering, repeated measurements, allocation, missingness and multiple planned comparisons as appropriate. Convert eligible independent units into a duration that includes relevant business cycles and outcome maturity. If sequential monitoring is required, use a designed sequential procedure with its own error and sample-size properties; do not repeatedly extend or stop an ordinary fixed-horizon test based on its p-value.