What is A/B testing, and how would you set up a test?

Example Answer

An A/B test compares a control experience with a treatment by randomly assigning eligible units, often users, to each group. I define the decision, primary metric, and guardrails before starting. Then I plan the sample size and duration, check the assignment and data quality, and compare the results using the agreed analysis.

Small example

For a checkout change, the primary metric might be completed purchases per assigned user. Guardrails could include errors and refunds. Keep each user’s assignment consistent and choose a randomization unit that fits possible interference, such as shared accounts.

Check for unexpected allocation imbalances and instrumentation problems before trusting the result. Avoid repeatedly checking a conventional fixed-horizon test and stopping when the p-value first looks favorable. Report the effect and uncertainty, and use a suitable sequential method if early stopping is required.

Reference

Microsoft Research: trustworthy experiment design, Practical guide to controlled experiments