Significant, and still wrong

Run a clean experiment, achieve statistical significance, and it feels like the work is finished. It isn’t. A p‑value answers one narrow question (could a gap this large have shown up by chance?) and then goes quiet on every question that actually decides whether you should act.

What will this earn? What happens if it fails? What does it cost to put in place, and can you walk it back? Significance is a gate, not a verdict. Treating it as the verdict is precisely how careful teams reason their way into expensive mistakes, feeling rigorous the whole way down.

The gap is widest in B2B. Sample sizes are small, sales cycles are long, and you often cannot correct a bad call quickly because you won’t even see the outcome for several quarters. The answer is decision discipline: making each result earn its decision rather than be waved through. It is exactly the habit most teams abandon in low-volume environments, where a single significant test sails past because gathering more data is slow.

ifour questions

The four questions significance leaves unanswered

Before any test result earns a decision, it has to survive four questions that no significance calculation will answer for you:

The expected return: what does acting earn, on average?
The downside if you’re wrong: not its probability, its size.
The cost to adopt and run: weighed against a lift this big, not against zero.
The reversibility: if it disappoints, how cheaply can you undo it?

There is a fifth question folded inside the first: not just whether the lift is positive, but whether it clears the bar that makes acting worthwhile at all, the win margin. That one is big enough to deserve its own treatment, so we hold it aside here and give it the whole of the next article, where it becomes one of the two questions the decision turns on.

Hold those in mind while we walk a single, ordinary decision through to the point where the statistics would have told you to sign.

iithe result

A result that looks like a win

Your current funnel converts 59 of 1,234 leads, or 4.8%. A vendor pitches a tool that, on a smaller pilot, converted 39 of 594, or 6.6%. The headline writes itself: +1.8 points, a clear improvement. Should you approve the contract?

Then look at what the headline hides. Treat the two funnels as what they are, true rates you are estimating from samples, not facts, and ask the question you actually care about: how likely is it that the vendor is genuinely better? About 94 chances in 100. Reassuring, until you read its complement: roughly one chance in eighteen that the tool is no better at all, or worse. And the believable range for the true lift runs from about −0.4 points to +4.3 points. Read that again: the range includes zero, and reaches into negative territory. The honest one-line summary is not “+1.8 points.” It is: somewhere between slightly worse and substantially better, and the pilot is too small to tell which.

Figure 1 draws that uncertainty as it really is: two distributions, one per funnel, each the full range of rates consistent with what was observed. Drag the slider under the chart to grow the pilot, and watch the one thing that actually narrows the doubt: a bigger sample.

The uncertainty behind the headline

Fig. 1 · interactive · drag to explore

Current funnel · 4.8% Vendor tool · 6.6%

Chance the vendor is truly higher

95% range for the true lift, points

Pilot size at this setting

Sample size ×1

Holds both observed rates fixed and asks: if you ran the same test at this multiple of the data, how much would the doubt shrink? At the real pilot size (×1) the curves overlap heavily, and that overlap is the risk.

How it’s computed. Each curve is a Beta posterior on that funnel’s true rate, starting with no assumption about the rate (a uniform prior): Beta(1+conversions, 1+misses). The chance the vendor is higher is exact numerical integration over the two; the lift range is the 2.5th to 97.5th percentile of the posterior difference. No fitted shapes, no decorative axes.

Figure 1 runs on this article’s pilot. If you have one of your own on the desk, you can run your own numbers in the A/B instrument: the same calculation, live, no setup.

iiithe single number

Why the single number lies

The pilot is small, so its range is wide. That is not a footnote to bolt on at the end; it is the result. A point estimate (“6.6%”, “+1.8 points”) is one draw from that range presented as if it were the truth. Build a decision on the single number and you have quietly assumed away the very thing that should worry you.

An average tells you where the middle sits. It says nothing about how far the floor can fall.

Suppose you make your peace with the uncertainty and accept the lift might be real. You are still not done, because the average profit with the tool against without it, the expected value, is necessary but not sufficient. Two decisions can share an expected value and be nothing alike: one where the worst case costs you a coffee, another where the worst case is the quarter. The average is blind to that. You have to look at the shape of the outcomes, not their mean.

ivthe downside

Pricing the downside

Make it concrete. Put the lift through a simple profit model over a five-year horizon. Each year, the extra profit is the lift in conversion, times the leads you process, times the rate at which qualified leads close, times the profit per customer, less the running cost. Accumulate across years, subtract the setup. Now feed it not the headline lift but the whole believable range of lifts from Figure 1, and let the model carry that uncertainty forward.

The same model in symbols

Pₙ = [ (MQLᵛ − MQLᵉ) × Pᶜ × L × SQL − Cᵝ ] × Yₙ − Cᵢ

MQLᵛ, MQLᵉ vendor vs. current conversion rate · L leads per year = 1,300
SQL sales-qualified close rate = 22% · Pᶜ profit per customer = €16,000
Cᵝ annual cost = €7,000 · Cᵢ setup cost = €25,000 · Yₙ years

The economics significance ignored

Fig. 2 · Monte Carlo · 20,000 draws

Cumulative profit, € median path break‑even

Median 5-year profit

Realistic downside, 5th percentile

Chance you’re underwater at year 5

How it’s computed. 20,000 lifts are drawn from the same posterior difference as Figure 1 (sampled exactly, not approximated) and each is pushed through the profit model above with every other input held at its stated value. The shaded band is the 5th to 95th and 25th to 75th percentiles of cumulative profit; the line is the median. Only the lift varies here; letting the rest of the system vary too is the next article.

vthe verdict

So, would you buy it?

If “+1.8 points, a clear improvement” was enough to earn your signature, that spread of outcomes should give you pause. Notice what changed and what didn’t: none of the statistics moved. The same pilot, the same lift, the same p‑value. What changed is that you finally priced the downside, and a meaningful slice of the believable range keeps you in the red for years, because the setup and running costs are certain while the lift is not. The readout under Figure 2 puts a live number on it: the chance you are still underwater at year five.

The four questions, restated now that they have teeth:

Not “is the lift positive” but what is the range of the lift.
Not “what’s the average return” but how bad is the bad case.
What does it cost to adopt, set against a lift this size.
And if it disappoints, how cheaply can you reverse it.

What this still leaves out. Plenty: opportunity cost, the time value of money, and the ideas you didn’t test because you spent the cycle on this one. And the B2B sting: a long sales cycle means you may not learn you were wrong for many quarters, which is the strongest reason of all to price the downside before you commit rather than after.

This piece let only the lift vary and held the rest of the business steady. The next one takes the most slippery question, is the lift big enough to matter, and makes it the whole question.

← All writing Try the instrumentThe Bayesian A/B test → Next in seriesBetter isn’t the decision. Better by enough is.