Bayesian A/B Testing: Decision-Making with Posterior Distributions

Research Team
1 Research Platform
Abstract —

This paper outlines a practical Bayesian framing for A/B tests, focusing on posterior probability and expected loss to guide go/no-go decisions in product experiments.

Keywords: bayesian statistics, experimentation, decision theory, product analytics

Why Bayesian A/B Testing?

Frequentist A/B tests are useful, but decision-making often needs answers like:

  • What is the probability that variant B beats variant A?
  • How large is the uplift likely to be?
  • What is the risk of shipping a change with small or negative effect?

A Bayesian approach provides these probabilities directly by modeling a posterior distribution over conversion rates.

Model Setup

Assume Bernoulli conversions with Beta priors:

θA∼Beta(αA,βA),θB∼Beta(αB,βB)\theta_A \sim \text{Beta}(\alpha_A, \beta_A), \quad \theta_B \sim \text{Beta}(\alpha_B, \beta_B)

After observing conversions xx out of nn exposures, the posterior is:

θ∣x,n∼Beta(α+x,β+n−x)\theta \mid x, n \sim \text{Beta}(\alpha + x, \beta + n - x)

Decision Metrics

Useful metrics derived from the posterior include:

  1. P(θB>θA)P(\theta_B > \theta_A), the probability of improvement.
  2. Expected uplift, E[θB−θA]E[\theta_B - \theta_A].
  3. Expected loss if selecting the wrong variant.

These map well to product decisions where the cost of a wrong launch is explicit.

Practical Guidance

  • Start with weakly informative priors when historical data is limited.
  • Favor decisions based on expected loss, not only on tail probabilities.
  • Report uncertainty intervals to communicate the range of plausible outcomes.

Conclusion

Bayesian A/B testing complements product decision-making by turning uncertain outcomes into probabilistic guidance. The framework scales from quick experiments to complex multi-variant testing.