Explain the role of control variables in A/B testing.
Instruction: Describe what control variables are, their importance in A/B testing, and how they can affect the outcomes of an experiment.
Updated
Official answer
“Control variable” can mean an implementation factor intentionally held constant or a measured covariate included in analysis. Distinguish these uses.
For a checkout experiment, keep unrelated features, logging definitions and intended eligibility rules comparable across variants. User age, device and prior activity can still vary across people. Valid random assignment balances such baseline characteristics in expectation; they do not all need to be made identical.
Prespecified stratification or adjustment for prognostic pretreatment covariates can improve precision. For example, adjust for prior purchase count when estimating purchases during the experiment, using inference appropriate to the design. Baseline adjustment does not guarantee exact balance or compensate for every implementation defect.
Avoid automatically controlling for a variable changed by treatment, such as pages viewed after assignment, when estimating the total effect. It can be a mediator or selection variable and can change or bias the comparison. Document the covariate timing, rationale and estimand before interpreting the adjusted result.