Beware false precision in forecasts and models
Treat any precise probability or quantitative forecast with explicit suspicion about whether the model fits the domain.
Key takeaways
- What it is: Treat any precise probability or quantitative forecast with explicit suspicion about whether the model fits the domain.
- Why it works: Numbers create a confidence effect: a model that says there is a 2.3% probability of failure feels more credible than "roughly 1 in 50" — even though both express the same probability and the precision may be entirely manufactured. In game-like domains, precision is justified; in wild-randomness domains, the precision is a property of the model’s assumptions, not of reality. Questioning precision redirects attention to model validity.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Dismissing any quantitative analysis because precision seems suspicious — the goal is to widen the confidence interval, not to abandon quantitative thinking.
Why it works
Numbers create a confidence effect: a model that says there is a 2.3% probability of failure feels more credible than "roughly 1 in 50" — even though both express the same probability and the precision may be entirely manufactured. In game-like domains, precision is justified; in wild-randomness domains, the precision is a property of the model’s assumptions, not of reality. Questioning precision redirects attention to model validity.
How to do it
- 1When you see a precise numerical forecast, ask: "What assumptions is this precision built on, and do those assumptions hold in this domain?"
- 2Re-express the forecast as a wide range: "This model says 2.3%, but given the assumptions, the real range could be 0.5%–15%."
- 3Weight the decision accordingly.
What the evidence says
ObservationalConsistent with evidence on model uncertainty and calibration in forecasting. Professional forecasters’ confidence intervals are systematically too narrow across domains from weather to finance, indicating that stated precision routinely exceeds warranted precision.
Honest caveat: Not all precision is false — in stable, well-understood domains with large data, model precision can be justified. The check is whether the domain warrants it.
Common mistake
Dismissing any quantitative analysis because precision seems suspicious — the goal is to widen the confidence interval, not to abandon quantitative thinking.
IX Coach prompts you to express any probability-based plan in range form rather than point estimates, exposing the model uncertainty before it drives false confidence.
Practice this with IX Coach →