Bayesian A/B Testing for Faster Decision-Making

Classical A/B testing relies on fixed sample sizes: you calculate the required visitor count in advance, wait until you reach it, compute statistical significance, then decide. This worked in the 2010s because traffic was expensive and tests could run for months. In 2026, performance marketing operates on weekly cycles, creative refresh happens every 14 days, campaign strategy shifts monthly. Running a landing page variant for 6 weeks isn't prudent—it's waste. Bayesian A/B testing solves this with sequential decision-making: your posterior distribution updates daily, and the moment you hit your confidence threshold, you stop the test and deploy the winner.

The Sample Size Trap in Frequentist Testing

Classical frequentist A/B testing hinges on p-value < 0.05. To reach this threshold, you run power analysis first: targeting 5% baseline conversion, 10% relative lift, and 80% statistical power requires a minimum of 3,100 users per variant. With 500 daily unique visitors, your test runs 12 days. The problem is this: by day 5, variant B is clearly winning—but statistical significance hasn't arrived yet, so you wait. By day 12, significance emerges, but your competitor has already launched a new landing page, your messaging feels stale. Frequentist testing inflicts dual damage: decide early and risk Type I error (false positive); decide late and incur opportunity cost. Sequential testing exists within the frequentist framework (Bonferroni correction, alpha spending functions), but it's complex. You must budget alpha for each interim analysis—to stop early, your critical value tightens. The result: tests lengthen or confidence drops.

Bayesian methods escape this bind because every observation is new information—yesterday's posterior becomes today's prior. Sample size isn't fixed; it's sequential. Each day your posterior updates, and the moment you see "probability that B outperforms A exceeds 95%," you stop and ship. Early stopping isn't a penalty—it's a feature.

Posterior Distribution and Sequential Updating

In Bayesian testing, you start with a prior distribution: your previous belief about the conversion rate. Testing an e-commerce landing page, you might set a baseline of 3% conversion with a standard deviation of 0.5% (based on historical data). This becomes a Beta(30, 970) prior. When the first 100 visitors arrive and variant B shows 4 conversions, the posterior updates like this:

Prior: Beta(α=30, β=970)
Likelihood: 4 successes, 96 failures
Posterior: Beta(α=30+4, β=970+96) = Beta(34, 1066)

Posterior mean = 34/(34+1066) = 0.0309 (3.09%). The next day brings 200 more visitors with 7 conversions. Yesterday's posterior becomes today's prior:

Prior: Beta(34, 1066)
Likelihood: 7 successes, 193 failures
Posterior: Beta(41, 1259)

Posterior mean = 0.0316 (3.16%). For variant A, over the same period: 500 visitors, 14 conversions. Posterior for A = Beta(44, 1456), mean = 0.0293. Now you compare the two posteriors: calculate P(B > A) using Monte Carlo simulation—draw 10,000 samples and count how often B exceeds A. If the probability is 73%, you're not confident yet. By day 5, when P(B > A) = 96%, you've hit your decision threshold (95%) and stop the test.

This is impossible in frequentist testing. Every interim peek risks alpha inflation and multiple comparison issues. Bayesian inference avoids this because updating is conditioned on likelihood—there's no requirement to fix sample size. Early stopping introduces no bias because the inference is sequential by design.

Practical Implementation: Stopping Rules and Thresholds

Bayesian A/B testing is easy to set up but requires disciplined stopping rules. Define three thresholds:

1. Minimum sample size (safety net): Prevents decisions that are too premature. Don't decide on fewer than 100 users per variant—posterior variance is wide, false positive risk is real. Google Optimize's 2019 whitepaper recommended a minimum of 250 conversions; in practice, 50–100 conversions per variant suffices (depending on prior strength).

2. Confidence threshold: P(B > A) > 0.95 is the classic choice. For aggressive decisions, use 0.90; for conservative tests, try 0.97. When financial impact is high (checkout flow changes), go with 0.99.

3. Practical significance (lift threshold): A 0.5% relative lift can be statistically significant but operationally meaningless. Add a practical threshold like lift > 5%. Calculate not just P(B > A), but P(B > A × 1.05).

Code example (Python + PyMC):

import pymc as pm
import numpy as np

# Prior: Beta(30, 970) — 3% baseline
with pm.Model() as model:
    p_A = pm.Beta("p_A", alpha=30, beta=970)
    p_B = pm.Beta("p_B", alpha=30, beta=970)
    
    # Observed data
    obs_A = pm.Binomial("obs_A", n=500, p=p_A, observed=14)
    obs_B = pm.Binomial("obs_B", n=500, p=p_B, observed=18)
    
    trace = pm.sample(5000, return_inferencedata=True)

# Posterior comparison
p_B_samples = trace.posterior["p_B"].values.flatten()
p_A_samples = trace.posterior["p_A"].values.flatten()
prob_B_better = np.mean(p_B_samples > p_A_samples)
prob_lift_5pct = np.mean(p_B_samples > p_A_samples * 1.05)

print(f"P(B > A) = {prob_B_better:.2%}")
print(f"P(B > A*1.05) = {prob_lift_5pct:.2%}")

Run this daily. Stop when prob_B_better > 0.95 AND prob_lift_5pct > 0.80. If these conditions are met by day 5, you've saved 7 days versus the frequentist's 12-day wait.

Tradeoff: Prior Selection and Sensitivity

The frequent criticism of Bayesian testing is that prior selection is subjective. A weak prior (Beta(1, 1)—uniform) lets the data speak purely, but convergence is slow. A strong prior (Beta(300, 9700)) lets historical knowledge dominate the posterior—new data has less impact. Balance is essential.

Prior selection strategy:

Test the prior's influence via sensitivity analysis: run the same data against Beta(1,1), Beta(10,990), and Beta(30,970). If posteriors diverge by more than 5%, the prior is too strong—choose a weaker one or gather more data.

Another tradeoff: Bayesian tests aren't as "publication-ready" as frequentist ones. Writing an academic paper requires a p-value; presenting to C-suite executives, a posterior plot suffices. In Conversion Rate Optimization workflows, speed is critical—within weekly sprints, Bayesian sequential testing delivers 40% faster results (per VWO's 2023 benchmark: 5 days median versus 8 days).

The Business Impact of Test Velocity

Bayesian sequential testing's real gain is velocity. In performance marketing, creative fatigue sets in at 10–14 days; campaign cycles last 30 days. If you close a landing page test in 12 days, you run 2 iterations per month. With Bayesian, you close in 5 days—6 iterations monthly. Assuming 5% lift per iteration, the annual compound effect is 12% under frequentist methodology versus 34% under Bayesian (1.05^12 vs 1.05^6).

Sequential testing multiplies gains in multivariate tests (A/B/C/D). Frequentist multiway comparisons apply Bonferroni correction, inflating sample size by 3–4×. Bayesian methods compute separate posteriors per variant and pairwise comparisons without alpha spending. With 4 variants, frequentist demands 15 days; Bayesian finishes in 6.

One more point: early stopping matters not just for winners but for losers. If variant B shows a 20% conversion drop, by day 3, P(A > B) reaches 99%—you stop and kill the test. Wasted traffic is prevented. Under frequentist logic, you'd wait the full 12 days, burning 9 days of underperforming traffic. Bayesian sequential testing provides downside protection.

Sequential Bayesian A/B testing is no longer a luxury—it's a necessity. Post-cookie deprecation, attribution is harder; campaign cycles are shorter; creative refresh moves faster. Classical frequentist tests can't keep pace. Bayesian posterior updates capture new information daily, and the decision threshold acts as your stop signal. Early stopping isn't bias—it's design. With disciplined prior selection, clear stopping rules, and a practical significance filter, Bayesian testing is both fast and reliable.