Describe how you would implement A/B testing for a machine learning model in production.

Example Answer

I would start by defining the decision we are trying to make. An A/B test is not just "new model versus old model." With valid randomization, measurement and analysis, it can estimate a causal effect on the chosen outcome for the tested population and setting while monitoring important guardrails.

From there, I would make sure randomization happens at the right unit, the exposure is logged correctly, and the treatment assignment is stable enough to avoid contamination. I would choose one primary success metric, a small set of guardrail metrics, and plan sample size and duration for adequate power at the effect size we care about. For ML systems, I also want segment cuts because models often help one cohort and hurt another.

Operationally, I would ramp carefully, monitor data quality and serving health during the test, and look for novelty effects before declaring success. Good experimentation discipline matters as much as model quality here.

Analysis boundary

Specify the outcome window and stopping rule before launch. A fixed-horizon test should not stop merely at the first favorable significance check; use an appropriate sequential design if repeated decisions are required. Check assignment counts, missing outcomes and sample-ratio mismatch. Analyze the assigned eligible population for the intention-to-treat effect rather than keeping only users who clicked or were successfully served. Exposure diagnostics explain failures but should not silently redefine the primary population.

Allow delayed outcomes to mature and report effect sizes with uncertainty. Distinguish planned subgroup tests from exploratory cuts, accounting for multiple comparisons where needed. A small health ramp can detect serving failures without being adequately powered to establish product benefit.

Common Poor Answer

A weak answer treats A/B testing like a checkbox, without talking about randomization, logging, sample size, guardrails, or cohort-specific impact.