How do you evaluate the effectiveness of a recommendation system using A/B testing?

Example Answer

I would state the decision first: for example, whether a new recommender improves completed purchases per eligible user without unacceptable latency, cancellations or harmful recommendations. I would pre-specify the primary metric, guardrails, eligibility rules, minimum effect worth detecting and analysis method.

I would assign users consistently to control and treatment, then verify assignment proportions, exposure logging and contamination. The unit of analysis must match the design; many impressions from one user are not independent users. Shared inventory or social effects may require a different randomization design.

Before launch, I would choose sample size and duration using the expected baseline, desired precision or power, and relevant weekly cycles. I would not keep checking a conventional fixed-horizon test and stop when its p-value becomes small. Early stopping requires an appropriate sequential design or a pre-defined safety rule.

At analysis, I would report the treatment effect and uncertainty on the primary metric, inspect guardrails, and investigate instrumentation or sample-ratio problems before interpreting the result. Click-through rate is a useful diagnostic but may reward clickbait; it is not automatically the business outcome. Segment analyses need a rationale and appropriate treatment of multiple comparisons.

A launch decision would consider effect size, uncertainty, operational costs and user outcomes. I would also watch for novelty and longer-term retention effects rather than assuming a short experiment establishes lasting value.

This is a proposed approach. Describe actual work and results only when they are your own.