When there is no dataset, the formal tools still need a number, and that number has to come out of someone’s head. Elicitation is the craft of getting it out well: a few deliberate questions that turn a person’s vague sense of an uncertain quantity into a calibrated range you can compute with. If heuristics & biases was the catalogue of how judgement goes wrong, this is how to get it to go right anyway.
The throughline is one idea. People are bad at giving a spread and an exact probability, the things the maths asks for directly (variances, point probabilities), and good at things the maths can use indirectly, concrete extremes and vivid comparison cases. So you ask for what they can answer, and do the arithmetic for them. The companion tool, Elicit, runs the core of it.
You have to estimate the conversion rate of a channel you have never run: no data, only judgement. We carry this one quantity through every step, from the first question to a distribution the tools can use. The numbers below are illustrative.
Six moves, each fixing a specific way the gut misfires.
Ask for three points, not an average and a spread
Do not ask for an average and a standard deviation; almost no one can give an honest second number that way. Ask instead for three points a person can actually picture (formally, quantiles): a low they would be surprised to fall below, a typical best guess, and a high they would be surprised to exceed. For the channel you might say: surprised below about 2 in 100, best guess about 5 in 100, surprised above about 11 in 100. Three concrete edges, not an abstract spread.
This is exactly what Elicit takes. It reads the three answers as the 10th, 50th and 90th percentiles and fits a calibrated curve through them. You describe the quantity in plain terms, a probability or rate (formally Beta), and the tool picks the curve.
Defeat the anchor
The first number said aloud owns the room: everyone else adjusts from it, and never far enough. That is the anchoring trap from Part I, and the defence is procedural. Never go round the table. Collect each person’s three points for the channel privately and at the same time, before anyone hears anyone else’s, then compare. Disagreement you can see is information; disagreement quietly averaged away is not.
Reference classes and base rates
Before adjusting for what is special about your case, start from how the wider class behaves: how do channels like this one usually convert? If comparable channels land around 4 in 100, that base rate should pull your typical far more than the story about why yours is different. Keeping the base rate written next to the estimate is half the cure for base-rate neglect.
Calibration training
Getting your 90% ranges to actually contain the truth about 9 times in 10, which forecasters call calibration, is a learnable skill rather than a fixed trait. The method is feedback: make ranged forecasts, record them, and later score how often the outcome fell inside the range. For the channel, the score is whether its true conversion rate eventually fell inside your 2-to-11-in-100 range. Most people start badly overconfident, their ranges far too narrow, and tighten up once they have seen their own hit rate. A calibrated estimator is worth more than a confident one.
Counterfactual stress and pre-mortems
Overconfidence shows up as intervals pinched too tight, so stress the ends. Take your high and assume the truth came in above it anyway: what would have had to be true? The question is easy to answer and usually surfaces a path you had ruled out, which pushes the high outward to where it belonged. Run the same move on the low: “what if the channel converts worse than 2 in 100?” might surface a seasonality you forgot. A pre-mortem, assume it failed and now explain why, is the same trick applied to a whole plan.
From one number to a distribution
Three good answers are still just three numbers. The last step turns them into a shape. Elicit fits a curve from the family that suits the quantity, here a probability or rate (formally Beta), and reports a fit quality: if your low, typical and high cannot all sit on one honest curve, that is the tool telling you the three are not internally consistent. Revise the numbers, or pick a family that fits.
A single fitted distribution is the unit the rest of the work runs on. Hand it to Simulate as one input among several and the channel’s uncertainty travels, undiluted, all the way to the bottom line. That is exactly how the two tools chain: Elicit’s curve becomes a Simulate input.
A calibrated range is not a hedge. It is the most useful thing a person who does not know can honestly give you.
None of this turns a guess into data. It makes a guess honest and computable: asked in the form people can answer, stripped of the first speaker’s anchor, tied to the base rate, widened to its real width, and handed on as a shape rather than a single hopeful number.