Ian Hargreaves.
Analysis that decides
Instrument № 03 · A/B Test
№ 03
A/B Test Instrument

Conversion
A/B

Run two campaigns, one converts higher, you call it the winner. But a higher rate in a small sample can be luck. This asks how often one campaign genuinely beats the other, and computes it exactly.
Campaign A
Observed: 4.8%
Campaign B
Observed: 6.6%
A win “by a meaningful margin” = B beats A by at least this many percentage points, not just a hair.
94%
Chance B beats A
87%
Chance B wins by the margin
+1.8
Most likely gap, points
…
Believable range of each campaign’s true rate AB
Method. Each true conversion rate is modelled as a Beta distribution (the two curves). Probabilities are computed exactly by numerical integration over those distributions: deterministic, no simulation, no random wobble. A uniform Beta(1,1) prior is used; with samples this size the data dominate it.
Method

Why not just
compare the rates?

03.1

A rate is a range

Six heads in ten flips doesn’t prove a biased coin. A campaign’s observed rate is one sample; the true rate sits somewhere in a range around it. Small samples mean wide ranges.

03.2

Compare the ranges

The two curves are those ranges. Where they overlap is the uncertainty. The question isn’t which rate is higher. It’s how much of B’s range sits above A’s.

03.3

Decide on the gap

“B beats A” isn’t enough if the gap is trivial. Set a threshold for what counts as a meaningful win, and read the probability of clearing it. That’s the number to act on.