Bayesian A/B Test Calculator
Drop in conversions and visitors for two variants and this reads the test as a decision rather than a verdict. It returns the posterior for each arm, the probability B beats A, the expected loss of shipping each one, a credible interval for the lift, and a straight answer to the only question that matters at the end of a test: can I stop now, and what does it cost me if I am wrong?
-
A, the controlB, the challenger
| If your threshold were | Leader's loss | Then today you would |
|---|
| Prior | P(B beats A) | Loss, ship leader | Lift | Verdict |
|---|
How this is computed
qbeta(). That detail is not cosmetic: a rule spread over the whole 0 to 1 interval walks straight past a narrow posterior and quietly returns zero.
✓ No data leaves your browser
✓ Verified against R by numeric integration and a 4,000,000 draw rbeta() simulation
✓ Free, no sign-up
P(B beats A) is not a stopping rule
It is the number everyone quotes and the number that decides the least.
P(B beats A) answers one question: which arm is ahead? It says nothing about by how much, and that omission is where money gets lost. Run enough traffic through two variants that are genuinely identical and P(B beats A) will wander; give it a rate difference of a hundredth of a percentage point and enough visitors and it will climb to 99% quite honestly. Neither situation is worth a deploy.
The opposite failure is just as common. A test sitting at P(B beats A) = 80% sounds unfinished, so teams keep it running. But if the entire plausible downside of shipping B is two hundredths of a percentage point, there is nothing left to learn that is worth the traffic. The test is done. It just does not look done through the lens of a probability.
Expected loss answers what P(B beats A) cannot: if I am wrong, what does being wrong cost? It is measured in conversion rate points, the same units as your business, which is exactly why you can compare it to a threshold and get a decision out.
What expected loss actually measures
An average over the worlds where your choice was the wrong one.
Ship B. Somewhere in the range of rates the data still leave open, A was really the better variant, and in those worlds you have given up the difference. Expected loss weighs every one of those worlds by how plausible it is and averages the shortfall:
Loss(ship B) = E[max(pA - pB, 0)] and Loss(ship A) = E[max(pB - pA, 0)]
Worlds where B really was better contribute nothing to the loss of shipping B, because you were right there. Only the regret counts. That is what makes the number small and interpretable: it is not "how wrong could I be", it is "how wrong am I likely to be, times how much that costs".
The two losses are not independent quantities. Subtract one from the other and everything uncertain cancels, leaving the plain difference in posterior means: Loss(ship A) - Loss(ship B) = mean(pB) - mean(pA). This tool checks that identity on every calculation, because it is a fact of arithmetic rather than a modelling choice, which makes it a very hard test to pass by accident.
Picking a threshold of caring
The one number you have to bring yourself. Choose it before you look.
The threshold of caring is the most conversion rate you are willing to leave on the table in order to stop today. Ship the leading arm once its expected loss falls below it, because at that point the plausible cost of a mistake is smaller than the amount you already declared you could shrug off.
0.1 percentage points is a reasonable default and the one loaded here. To do better than a default, work from the money. A checkout page at a 3% conversion rate taking 200,000 visitors a year, at 40 dollars of margin per conversion, moves about 800 dollars a year per hundredth of a percentage point. If a 1,000 dollar annual swing is beneath your notice, your threshold is somewhere near 0.01pp. Set it far below what you care about and the test runs forever chasing a difference that would never change a decision.
Two failure modes to avoid. Setting the threshold after seeing the losses is just choosing the answer you wanted. And a threshold of zero never triggers: expected loss reaches zero only with infinite data, so demanding certainty is the same as demanding the test never ends.
A worked example: when to stop
One test, four checkpoints, a threshold of 0.1pp fixed before any data arrived.
Why peeking is fine here, and what still bites
The honest version, which is not "Bayesian methods make peeking free".
Looking is genuinely free. A posterior is a summary of the data you hold; it does not know or care whether you looked yesterday. There is no multiple-comparisons correction to apply, no alpha to spend, and nothing to recompute because you glanced at a dashboard. This is a real difference from a fixed-horizon frequentist test, where the p-value's guarantee is tied to a sample size you promised in advance.
The part that still bites: a stopping rule is not a look. "Stop the moment expected loss dips below the threshold" gives every random wobble a fresh chance to end the test at a flattering moment, and it does raise how often you ship the worse arm compared with running to a fixed horizon. Anyone telling you optional stopping is completely free under a Bayesian framing is selling something.
What rescues it is that the loss framing bounds the damage rather than eliminating it. You only stop when the expected cost of being wrong is under your threshold, so a wrong call still costs less than the amount you said you could live with. That is a much more useful guarantee than "the error rate is 5% if you followed the protocol you did not follow". If you want the tighter version, fix the traffic budget in advance and read the loss at the end; the number means the same thing either way.
Two things the model genuinely will not save you from, since no amount of arithmetic can: a broken randomiser that splits traffic unevenly, and a novelty effect where B wins for a fortnight because it is new. Both break the assumption that every visitor in an arm is an exchangeable draw from that arm's rate, and both look exactly like a real lift from in here.
What this tool does and does not do
It answers a decision question. For the significance question, the frequentist tool next door is the right one.
| Question | Where it is answered |
|---|---|
| Which arm should I ship, and can I stop now? | Here. Expected loss against a threshold of caring. |
| Is the difference statistically significant? What is the p-value? | Deliberately absent. That is a different question with a different logic, and the A/B Test Calculator answers it properly, with the z-test, the confidence interval and the sample-size planner. |
| How strongly do the data favour "there is a difference" over "there is none"? | Not here. That is a Bayes factor, and it answers an evidence question, not a decision one. Use the Bayes Factor Calculator. This page stays on expected loss on purpose: a decision needs units you can compare to a business threshold, and an evidence ratio has none. |
| What does a Beta posterior even look like? What are alpha and beta? | The Beta Distribution Calculator takes one distribution apart properly. |
| How many visitors will I need? | Plan it with the Sample Size Calculator for Proportions. Useful even here: a loss-threshold rule still needs a traffic budget you are willing to spend. |
| More than two variants | Not supported. Two arms only. Comparing three or more needs P(best) across every arm and a loss against the best of the rest, which is a genuinely different calculation and not a loop over pairs. |
Frequently asked questions
What is a Bayesian A/B test?
It treats each variant's conversion rate as an unknown quantity carrying a whole distribution of plausible values, rather than a fixed number to be tested against a null hypothesis. Start from a Beta(a, b) prior, add the conversions to a and the non-conversions to b, and the posterior is Beta(a + conversions, b + non-conversions). Every output on this page is read off those two posteriors.
What is expected loss, in one sentence?
The average conversion rate you give up by shipping a variant, counted only in the scenarios where that choice turns out to be wrong, and weighted by how plausible each scenario is. It is in percentage points, which is what makes it comparable to a business threshold instead of just being another probability.
Is P(B beats A) a good stopping rule?
No. It tells you which arm is ahead and never by how much, so it hits 99% for a difference too small to be worth a deploy and sits at 80% when the remaining downside is already negligible. Stop on expected loss instead.
How do I pick the threshold of caring?
Work from the money: multiply your traffic by your margin per conversion to find what a hundredth of a percentage point is worth per year, then set the threshold at the point where you would genuinely shrug. 0.1pp is a sane default. Pick it before you look at the data, and never set it to zero, because expected loss only reaches zero with infinite traffic.
Can I peek at the results?
Looking is free: the posterior is just a summary of the data in hand. Stopping on a peek is not free. Ending the test the instant loss dips below the threshold does ship the worse arm more often than running to a fixed horizon would, because wobbles get repeated chances to end things at a flattering moment. The loss rule bounds the damage rather than removing it, which is the honest claim.
Which prior should I use?
Uniform Beta(1,1) unless you have a reason. It adds one imaginary conversion and one imaginary non-conversion per arm and is swamped within a few hundred visitors. Jeffreys Beta(0.5,0.5) is the standard low-information alternative and behaves better at very low rates. Use a custom informative prior only when you can say where it came from, then run the prior sensitivity mode to see how much of the answer it is responsible for.
Why do the two expected losses not add up to the lift?
They do, once you subtract instead of adding: Loss(ship A) - Loss(ship B) is exactly mean(pB) - mean(pA). The uncertainty cancels because a positive part minus a negative part reconstructs the whole. This tool tests that identity on every calculation.
Why does the expected loss read "<0.0001pp"?
Because it is genuinely negligible, not because the calculation failed. When one arm is far ahead the loss of shipping it collapses fast: for one of the verified cases here the true value is about 8 times 10 to the power of minus 84. No decision could use a number like that, and ordinary double precision cannot even represent it accurately, so rather than print a digit that would be noise the tool reports a bound. Any threshold of caring you could sensibly set is thousands of times larger.
Does this calculator match R?
Yes, and by more than one route. Posterior means, quantiles, P(B beats A), both expected losses and the lift intervals are checked against R over 20 cases by fine numeric integration and, independently, by a four million draw rbeta() Monte Carlo simulation. The simulation is the important one: it tests the derivation rather than the arithmetic, since integrating my own formula would only ever confirm my own algebra. The smallest values are settled by an exact finite-sum calculation carried at 80 decimal digits, because below about a billionth R's own integration is no longer accurate either.
Go deeper
The neighbouring tools and the theory behind this one.