A/B Test Calculator
An A/B test compares two versions of something, like two button colors on a page, to tell whether one really performs better or the gap is just random luck. Drop in your numbers to plan how many visitors you need, or to check whether a result you already have is real. Everything runs in your browser.
iNew to A/B testing? Read the 4-minute primer
What it is. An A/B test routes traffic to two variants of the same thing (page, email, button), measures a binary outcome (converted? clicked?), and asks whether B's true conversion rate differs from A's, or whether the gap you saw is the kind of jitter you would expect from random sampling.
How to read it. Four numbers do most of the work. Lift is B minus A in percentage points (or relative percent). p-value is the chance of seeing a gap this big when there really is no difference: small p means the gap is hard to dismiss as noise. 95% CI for the lift is the range of differences the data are compatible with: if it crosses zero, you cannot rule out "no effect". BF10 is a Bayes factor: above 10 is strong evidence for B, below 1/10 strong evidence for A.
Which mode. Not run yet? Plan sizes the experiment from a baseline rate, a minimum lift, and your power. Already done? Analyze takes the per-arm visitors and conversions; the framing toggle (frequentist, Bayesian, both) picks which numbers show. Frequentist pre-sets alpha and power and decides on p below alpha; Bayesian updates each arm to a Beta posterior and reports P(B > A). Both answer the same question with different assumptions.
-
How this is computed
✓ No data leaves your browser
✓ Verified against R's prop.test(), pwr.2p.test() and integrate()
✓ Free, no sign-up
Anatomy of an A/B test
Every number this tool shows comes from one of these five pieces. No simulation: the Bayesian quantities are exact numerical integrals, so they are identical on every run.
Two-proportion z-test (frequentist): p_pool = (c_a + c_b) / (n_a + n_b) SE_0 = sqrt( p_pool (1 - p_pool) (1/n_a + 1/n_b) ) z = (p_b - p_a) / SE_0 p = 2 (1 - Phi(|z|))
Two-proportion z-test. Pool the two arms under H0 to estimate one shared rate, compute the standard error from it, and read off a z-statistic. The CI for the lift uses the unpooled SE (each arm's own variance) so it centers on the observed gap. No continuity correction by default. Matches prop.test(c(c_a, c_b), c(n_a, n_b), correct = FALSE) exactly.
Bayesian beta-binomial conjugate:
prior: p_a, p_b ~ Beta(alpha_0, beta_0)
posterior: p_a ~ Beta(alpha_0 + c_a, beta_0 + n_a - c_a)
p_b ~ Beta(alpha_0 + c_b, beta_0 + n_b - c_b)
P(p_b > p_a) = integral dbeta(x; B) pbeta(x; A) dx
BF10 = B(a1,b1) B(a2,b2) / [ B(a0,b0) B(a0+c_a+c_b, b0+n_a+n_b-c_a-c_b) ]
Beta-binomial conjugate. A Beta prior is conjugate to the Binomial likelihood, so each posterior is Beta in closed form. We integrate the two posteriors for P(p_b > p_a), integrate the distribution of the lift for the 95% credible interval, and get BF10 (rates differ vs equal) in closed form from Beta functions. With a Beta(1,1) prior the posterior mean is (c+1)/(n+2). Matches R's integrate() and lbeta(). BF10 and P(B>A) answer different questions: a high P(B>A) with BF10 near 1 means B is probably ahead but the data barely favour "different" over "equal".
Sequential looks (Pocock boundary): K equally spaced interim looks constant boundary c_K holds overall alpha per-look nominal alpha = 2(1 - Phi(c_K)) declare significance if |z| > c_K at any look
Sequential alpha-spending. Peeking K times and stopping early on a single significant look pushes your real false-positive rate far above alpha. A Pocock design uses one constant z-boundary c_K at every look, chosen so the family-wise error stays at alpha; the per-look nominal alpha is then 2(1 - Phi(c_K)). This mode reads c_K from the tabulated values in Jennison & Turnbull (2000), Table 2.1 (e.g. c_K = 2.289 for K = 3 at alpha = 0.05, a per-look alpha near 0.022). These come from the group-sequential literature, not a base-R function; for a full design use gsDesign.
Sample size (two-prop, arcsine / Cohen's h): h = |2(arcsin sqrt(p2) - arcsin sqrt(p1))| n_per_arm ~ 2(z_alpha + z_beta)^2 / h^2 (pwr.2p.test solves the exact power equation)
Sample size for proportions. The arcsine transform stabilizes variance so both arms contribute equally; that is Cohen's h. The two-sample noncentrality is h*sqrt(n/2), giving n per arm near 2(z_alpha + z_beta)^2 / h^2. This tool returns the value from pwr::pwr.2p.test(), which solves the full power equation, so it can differ from the closed form by a visitor or two. It assumes 50/50 allocation, the most powerful split for a fixed total; for a deliberately unequal split use pwr::pwr.2p2n.test().
Lift, RR, and absolute difference: absolute lift = p_b - p_a (percentage points) relative lift = (p_b - p_a) / p_a risk ratio = p_b / p_a
Lift vs RR vs absolute difference. Be explicit. Saying "B is 20% better" is ambiguous: a baseline of 10% rising to 12% is +2pp absolute, +20% relative, RR = 1.20. Confidence intervals and BF10 are computed on the absolute scale here; the report line shows the relative figure for stakeholder summaries.
When this is the wrong tool
The two-proportion / beta-binomial pair fits one binary outcome per user in two arms. Reach for something else when your data breaks that shape.
| If you have | Use instead |
|---|---|
| Three or more variants | A multi-armed bandit (Thompson sampling), or a Bonferroni / Holm correction over pairwise z-tests. |
| Continuous outcome (revenue, time on page) | A Welch two-sample t-test. The two-proportion test forces a binary outcome. |
| Survival or time-to-event outcome | Kaplan-Meier with a log-rank test, or a Cox model. Censoring breaks the binomial assumption. |
| Repeated measures on the same user | A paired test (McNemar for binary), or a mixed-effects model with a user random intercept. |
| Ratio metrics (clicks per visit) | Delta-method or bootstrap CI for the ratio. Each session is not IID because the denominator is itself random. |
| You peeked early and stopped on a hit | The reported p is wrong. Use a group-sequential design up front, or always-valid intervals (mSPRT). |
Frequently asked questions
When should I use a two-proportion z-test versus a Bayesian beta-binomial?
Use the two-proportion z-test when stakeholders expect a p-value or your team operates under a fixed alpha. Use the Bayesian beta-binomial when you need a direct probability statement like "B beats A with 94% probability" or want to incorporate a prior. The calculator runs both side by side (the Both framing) so you can compare verdicts before deciding which to report.
How do I calculate the sample size needed for an A/B test?
Switch to Plan mode and enter your baseline conversion rate, the minimum detectable effect (MDE) you care about, your target power (typically 0.80) and significance level (0.05). The calculator returns the required sample size per arm using the same formula as R's pwr.2p.test. For a 10% baseline and a 1pp MDE at 80% power, you need about 14,750 per arm.
Can I peek at A/B test results before the test ends?
Not safely with a fixed-horizon test: peeking inflates the Type I error. Use the Sequential mode, which applies a Pocock constant boundary so you can take K interim looks while keeping the family-wise error rate at your alpha. With 3 looks at alpha = 0.05, the per-peek boundary rises to |z| > 2.289 and the per-look nominal alpha shrinks to about 0.022.
What is the difference between absolute and relative lift?
Absolute lift is B minus A in percentage points: 10% to 12% is +2pp. Relative lift divides by the baseline: the same change is +20% relative. Risk ratio is the plain ratio, 1.20. Confidence intervals and the Bayes factor here are on the absolute scale; always state which one you mean when you report a result.
Does this calculator match R?
Yes. Every displayed number (z, p, lift CI, sample size, P(B>A), the credible interval, BF10 and the Pocock boundary) is checked against R's prop.test(), pwr.2p.test() and integrate() across 29 cases to a relative tolerance of 1e-6.
Go deeper
- A/B testing in R, end to end - design, analysis and reporting in one tutorial.
- Hypothesis testing in R - the conceptual underpinnings of frequentist inference.
- Power analysis in R - how MDE, alpha and power trade off.
- Proportion tests in R - prop.test, binom.test and when to pick which.
- Confidence interval calculator - if you only need the CI for the lift.
- Power analysis tool - sample size for means, proportions and correlations.
Numerical accuracy: pnorm/qnorm near machine precision; the Bayesian quantities are deterministic (numerical integration of the Beta posteriors and closed-form Beta functions, no random sampling). Every displayed number is checked against R's prop.test(), pwr.2p.test() and integrate() across 29 cases to a relative tolerance of 1e-6. Pocock constants are the tabulated values from Jennison & Turnbull (2000).