Practice probability calibration
Regularly make probabilistic predictions and track your accuracy across many of them.
Key takeaways
- What it is: Regularly make probabilistic predictions and track your accuracy across many of them.
- Why it works: Calibration is the match between your stated confidence and your actual accuracy: if you say "80% confident" across 100 predictions, you should be right roughly 80 times. Hindsight bias interferes with calibration by making you believe you were more confident about outcomes you got right than you actually were. Training calibration directly — making many small forecasts and scoring them — builds accurate self-knowledge about uncertainty, which is the foundation of better decisions under uncertainty.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Making predictions only on things you expect to get right, rather than sampling broadly, which produces fake calibration from a non-random selection of easy cases.
Why it works
Calibration is the match between your stated confidence and your actual accuracy: if you say "80% confident" across 100 predictions, you should be right roughly 80 times. Hindsight bias interferes with calibration by making you believe you were more confident about outcomes you got right than you actually were. Training calibration directly — making many small forecasts and scoring them — builds accurate self-knowledge about uncertainty, which is the foundation of better decisions under uncertainty.
How to do it
- 1Pick a domain with checkable outcomes: sports results, project deadlines, colleague decisions.
- 2Make a probabilistic prediction and write it down with a confidence level.
- 3After the outcome, score whether the outcome fell within your stated probability.
- 4Over dozens of predictions, calculate your calibration score and notice where you are systematically over- or under-confident.
What the evidence says
ObservationalCalibration training is one of the few debiasing interventions with real evidence of improving forecasting accuracy. Research on superforecasters found that people trained in probabilistic thinking outperform experts without such training.
Honest caveat: Superforecasting findings come from specific geopolitical forecasting contexts; how much the skill transfers to personal and professional decisions is plausible but not directly studied.
- — Tetlock & Gardner (2015), Superforecasting: The Art and Science of Prediction
Common mistake
Making predictions only on things you expect to get right, rather than sampling broadly, which produces fake calibration from a non-random selection of easy cases.
IX Coach logs your predictions, tracks outcomes, and builds your personal calibration profile — showing where your confidence reliably leads or lags your accuracy.
Practice this with IX Coach →