What is the importance of randomization in an A/B test?

Official answer

Random assignment makes treatment allocation independent of potential outcomes under the design. It prevents systematic selection into treatment based on user characteristics and balances measured and unmeasured baseline factors in expectation. A particular finite sample can still be imbalanced.

Random assignment and random sampling are different. Randomizing eligible volunteers does not automatically make them representative of every user. The causal estimand initially concerns the experiment's eligible population and implementation.

Use a stable assignment unit appropriate to spillovers, preserve assignment across visits, and analyze outcomes using the intended assignment. Check allocation, exposure and logging. Interference, differential missingness, noncompliance or instrumentation defects can undermine the intended comparison.

For example, randomly allocating accounts between two checkout designs supports an intention-to-treat comparison of purchases per assigned account. Restricting analysis to accounts that completed checkout would select on an outcome affected by treatment. Randomization is the basis of the comparison, not a guarantee that every subsequent analysis is valid.