Score your own past predictions to calibrate your outside-view use
Keep a forecast log and score it — you cannot improve calibration without feedback on where you were over- or under-confident.
Key takeaways
- What it is: Keep a forecast log and score it — you cannot improve calibration without feedback on where you were over- or under-confident.
- Why it works: Calibration is the alignment between stated confidence and actual accuracy. Most people never find out how well-calibrated they are because they do not track their predictions. Without this feedback loop, outside-view adoption remains a good intention rather than a skill. Scoring forces the system to produce legible error signals and creates the conditions for genuine improvement.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Recording predictions in natural language rather than explicit probabilities, making retrospective calibration impossible — "I thought it was likely" cannot be scored.
Why it works
Calibration is the alignment between stated confidence and actual accuracy. Most people never find out how well-calibrated they are because they do not track their predictions. Without this feedback loop, outside-view adoption remains a good intention rather than a skill. Scoring forces the system to produce legible error signals and creates the conditions for genuine improvement.
How to do it
- 1Start recording significant predictions with probability estimates: "70% chance this takes less than three months."
- 2Set a calendar reminder to record the outcome when it resolves.
- 3Compute your calibration over time: for everything you said had a 70% probability, did roughly 70% happen?
- 4Identify domains where you are systematically over- or under-confident and adjust your base-rate anchoring accordingly.
What the evidence says
ObservationalTetlock’s forecasting tournament research found that tracking and scoring predictions is the feedback mechanism that separates superforecasters from the rest — those who tracked and reflected improved substantially, while those who did not stagnated.
Honest caveat: Calibration scoring requires a sufficient volume of scored predictions to be statistically meaningful; individual life decisions are not resolved frequently enough to produce clean calibration curves in the short term.
- — Tetlock & Gardner (2015), Superforecasting
Common mistake
Recording predictions in natural language rather than explicit probabilities, making retrospective calibration impossible — "I thought it was likely" cannot be scored.
IX Coach maintains a scored prediction log across your coaching sessions, providing the feedback signal that makes outside-view thinking a genuine skill rather than an abstract principle.
Practice this with IX Coach →