statistical significance vs practical significance a/b testing

A rate is one measurement, not the truth

A quick change of example: the vendor contract steps aside for now, because the “better by enough” logic is cleanest on a plain A/B test, and the final piece brings the vendor case back with the whole system varying. Start with a single conversion rate. 30 conversions from 1,000 visitors reads as 3.0%, and it is tempting to file that as a fact about the world. It isn’t. It is one photograph of a moving thing. Run the identical campaign again, change nothing, and you might log 28 the next time, or 33. Not because anything shifted, but because conversion is a coin-flip repeated a thousand times and the count lands a little differently each run.

So the honest object is not a single number. It is the band of rates the number is consistent with. A small sample gives a wide band and little confidence; a large one a narrow band and a lot. The lone figure on the dashboard is a lie of omission: accurate as far as it reaches, and silent about how far that is.

From one range to two

Now the comparison everyone actually runs. Version A converts 30 of 1,000, or 3.0%. Version B converts 40 of 1,000, or 4.0%. B is a point ahead, and the random split does its job: the only systematic difference between the two groups is the thing you changed, so a real gap is genuinely caused by the change, not smuggled in by some confound. That logic is clean. It is also only half the argument.

Because each rate is a band, not a point. Draw both bands and the question sharpens from the lazy version, is B’s number bigger? (it is, trivially), to the one that matters: how much of B’s band sits clear of A’s? The more the two bands overlap, the more room there is for the variants to be really level and for the gap you saw to be the dice; the exact chance comes from the distribution of the true gap, B minus A, in Fig. 1 below.

A win can be real and still be worthless

Say the chance B is genuinely higher comes out at roughly 89 in 100. B is almost certainly the better variant. Reassuring, until you ask what the win is actually worth. Suppose acting on it means €30,000 of engineering, retraining and a switching cost you can’t recover, and the genuine lift, real as it is, returns about €400 a year. You will have made a confident, evidence-backed, thoroughly rigorous decision to lose money.

“Is B better?” is a question about the test. “Is B worth it?” is a question about your business, and your test is blind to that.

The fix is a single piece of discipline: the bar, set before you look at the result. It is the smallest lift that would actually pay for the change. For a free, reversible tweak, a subject line or a button colour, the bar sits near zero; ship on a hair, you lose nothing if you’re wrong. For a signed annual contract with a setup fee, the bar might be a full percentage point or more before the change is worth making at all. What makes the bar honest is the timing. A bar you fix before the data is a commitment. A bar you fix after seeing the number is a story you tell yourself to license the thing you’d already decided to buy.

Better, by enough

Put the bar at one point, so B must beat A by at least +1.0 to clear the cost. The most likely lift lands right on it, about +1.0, but the believable range runs from roughly −0.6 to +2.6 points, so the chance B truly clears a full point is close to a coin toss. Hold the two numbers side by side: about 89 in 100 that B is better at all, and about 50 in 100, a coin toss, that it is better by enough to act. Both true, same data, same test. A report that prints only the first is quietly recommending you spend €30,000 on the outcome of a flipped coin.

Here is the whole decision in one instrument, and it is worth a slow read. The top panel is the two ranges again, A and B, each as wide as your evidence is thin; the more they overlap, the more room there is for the variants to be really level. Difference the two, and you get the bottom panel, which turns that room into an exact chance: a single distribution of the true gap, B minus A, with the bar dropped in. Two knobs sit underneath. Sample size is how much data you have gathered, slide it up and every band tightens, the overlap closes, the gap sharpens. The bar is how big a lift you would need before acting is worth it, slide it right and watch the shaded mass that clears it fall away. Two readouts move together, the chance B is better at all and the chance it is better by enough, and the distance between them is the decision most dashboards never show you.

The two questions under every number

Strip away the specifics and the same skeleton is under every metric on every dashboard: the figure you’re shown is one draw from a distribution you can’t see. Deciding well means asking two questions, in order.

• How sure am I the difference is real? How much of the better variant’s range sits clear of the other’s.
• Is it big enough to be worth the cost of acting? How much of that range clears the bar the change has to pay for.

Most reporting answers the first and lets you assume it has answered the second. “Did it go up?” is the question the dashboard is built for. “Is it up by enough to pay for chasing it?” is the question the decision is built for. They are not the same question, and the distance between them is exactly where good analysis quietly turns into an expensive mistake.