How can Bayesian methods be applied in A/B testing?

Example Answer

For a binary conversion metric, I can model each variant’s conversion rate with a Beta prior and update it with a binomial likelihood. The posterior then describes uncertainty about the rates under that model. I can estimate the probability that B beats A, the probability that its lift exceeds a practical threshold, and the expected loss from choosing either variant.

I would set the prior, decision rule, minimum exposure, and maximum duration before launch. A Bayesian analysis can support a sequential decision policy, but repeatedly changing the metric or population does not become harmless because we use a posterior. I would check prior sensitivity and simulate the decision rule under realistic traffic and effect sizes. The choice to ship should reflect both uncertainty and business cost, not just a high probability of a tiny positive lift.

Worked example

Hypothetical independent Beta(1, 1) priors: A with 20 conversions in 100 trials has posterior Beta(21, 81); B with 30 in 100 has Beta(31, 71). Posterior draws can estimate P(p_B − p_A > minimum_lift).

Why this works

It explains a concrete Bayesian update and treats stopping as a decision-policy design problem rather than promising free certainty.