How would you use A/B testing to evaluate a new feature?

Example Answer

I would start with a hypothesis, one main outcome, and guardrails such as errors or repeat use. Then I would choose an appropriate randomization unit and work with analytics on the sample size and analysis plan before looking at results. Users or accounts should stay in their assigned variant when that fits the experiment.

Before interpreting any lift, I would check exposure logging and whether the observed allocation matches the intended split. I would use the agreed analysis window and report the effect size with uncertainty, rather than repeatedly stopping when a favorable result appears. A positive result still needs to be useful enough to justify the change and acceptable on the guardrails. If the test is inconclusive or the setup is broken, I would say that clearly and decide whether to investigate, redesign, or gather more evidence.

Make it your own

Choose a specific feature, randomization unit, primary metric, and guardrail. Explain the business decision the experiment supports. Follow the analysis method your team has agreed on; a fixed-horizon test and a valid sequential method have different stopping rules.

Why this works

It distinguishes experiment design, data quality, statistical evidence, and the product decision.

Follow-up to practice

What would you do if the treatment and control user counts were unexpectedly different?