sensitivity analysis vs monte carlo simulation

When the equation has too many moving parts

The profit model from the last article had one uncertain input, the conversion lift, and held lead volume, close rate and profit per customer at tidy point values. That is fine for isolating the lift. It is dangerous as a basis for signing. A vendor can deliver exactly the conversion improvement promised and still lose you money, if the leads thin out, or the close rate slips, or the average deal shrinks. Not because anything went wrong, but because those numbers were always ranges and you happened to land low on several at once.

The comparative question, is the vendor better than us?, can be answered while ignoring all of that, because the unknowns mostly cancel between the two options being compared (the two arms of the test). The absolute question, will this contract make money?, cannot. It needs lead volume, close rate and deal size carried as uncertainty, together, all the way to the bottom line.

Sensitivity: one input at a time

Before reaching for anything clever, do the cheap thing first. Take the five-year profit model from the first article and give every input not a single value but a low, a middle and a high. Then wiggle them one at a time: hold everything at its middle, push a single input down to its low and up to its high, and watch how far the bottom line swings. That is a sensitivity analysis; drawn as one bar per input, top to bottom through the funnel, it is a tornado.

Two things jump out of the chart below, and you can push them around. The lift dominates by a distance, and it is the only bar that crosses zero: land low on the lift and no amount of good fortune downstream brings you back. Deal size and close rate matter, but they only ever scale a result whose sign the lift has already decided. Widen any input’s range and its bar stretches, but the lift has to be pinned down a long way before deal size or close rate competes with it.

And the tornado earns its keep: you can build one in Excel in a few minutes, no code and no sampling, and for a simple, low-stakes decision it is often all you need. The catch is what it cannot tell you. It is honest about which input matters and silent about almost everything else. Read that bottom strip: every input at its low together, then its high together, gives a range so wide it is useless, a six-figure loss at one end and several times the base case at the other. But all-low-at-once and all-high-at-once are corners the real world almost never visits, because leads, close rate and deal size do not conspire to fail on the same day. The tornado has no way to say how likely any combination is, so it cannot answer the question that matters most for a signature: what is the chance you lose money?

Simulation: every input at once

So stop poking the model at its edges and start sampling it. Draw one plausible value for every uncertain input at the same time, each from its full range rather than its three corners, run the complete set through the equation, and record the profit. Do it tens of thousands of times. The pile of answers is the distribution you could not write down, and unlike the corners, the middle of that pile is where the real world actually spends its time.

That is Monte Carlo, built in the 1940s by physicists who faced systems too tangled to solve but easy to play out. Where the tornado moves one input and freezes the rest, the simulation lets them all move together, so compounding shows up on its own: several uncertain factors multiplied spread their doubt rather than averaging it, and the result leans skewed. The picture below builds it in stages, the three wobbling inputs as the curves they really are, and underneath, the single profit distribution that falls out when you sample all of them at once. Sharpen or blur any input with its slider and watch the profit curve, and the chance of loss, answer.

The five-year decision

The distribution above is the answer at year five. Trace it year by year on the real decision, the vendor case from the first article, current funnel 59/1,234 against the pilot’s 39/594, every input now a range drawn fresh for each of forty thousand runs, and two structural facts fall straight out of the arithmetic. First: a negative lift cannot be rescued. The downstream factors are multipliers, so a strong year for deal size or close rate scales a real gain up, but it scales a real loss up just as faithfully. Second, and more hopeful: a thin but positive lift can be carried over the line by an above-average year downstream. Good conditions rescue marginal wins; they never rescue genuine losses.

Read the lower strip carefully, because it holds the whole argument. The chance of being underwater falls steeply over the first few years, then stops falling and settles on a floor; the right-hand readout under the chart gives the live floor. The early drop is just setup cost being recouped: give it enough years and you almost always clear a fixed €25,000. The floor is something time cannot touch, the share of futures in which the vendor’s true lift was simply too small to cover its own running cost. A longer horizon fixes the cost-recoupment problem. It does nothing for a bad lift: if the tool was never really better, you are underwater forever.

Time cures the cost of starting. It cannot cure a thing that doesn’t work.

Two questions worth more than a point estimate

A point forecast answers a question nobody should be asking, “what is the single most likely outcome?”, and hides the only two that matter for a commitment you can’t easily unwind:

• What is the probability we lose money? Not the average case, the share of futures that end in the red, and the floor that share settles on.
• Which input, measured better, would move that probability most? That is where the next pound of analysis should go, the variable whose range, narrowed, shrinks the downside fastest.

That second question is the quiet payoff of building the simulation at all. Once the whole system is in front of you as a distribution, you can ask it not just “how risky is this?” but “what should I go and learn to make it less risky?”, and spend your next measurement where it actually buys down doubt, rather than on whatever was easiest to count. The instrument above shows it directly: sharpen the lift and the loss probability drops hardest, because that is the input the whole outcome hangs on.

Read in sequence, the series is one frame seen at three depths. ‘Is it real?’ and ‘is it better by enough?’ are the questions the first two pieces turn on; once the whole system moves at once they collapse into the one this piece set out to answer, how often do we lose money. And that finally settles the last of Part I’s four questions, the one left open until now: reversibility. A signed, multi-year contract is the low-reversibility case, you cannot cheaply unwind it, so it is the floor, the probability of loss, that governs the call here and not the median. When you can walk a decision back for the price of a coffee the average is enough; when you cannot, you have to price the bad-lift futures before you sign, not discover them after.