Track your estimates and calibrate
Compare your Fermi estimates to actual figures when you can, and use the gap to improve future estimates.
Key takeaways
- What it is: Compare your Fermi estimates to actual figures when you can, and use the gap to improve future estimates.
- Why it works: Fermi estimation skill is built through feedback loops — comparing estimates to actual values reveals systematic biases (e.g., consistently underestimating population or overestimating rates) that can be corrected with awareness. Without feedback, estimation skill plateaus at the level of intuition; with feedback, it improves measurably toward expert calibration.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Checking an estimate against reality once and concluding you are well calibrated — calibration is a statistical property visible only across many predictions.
Why it works
Fermi estimation skill is built through feedback loops — comparing estimates to actual values reveals systematic biases (e.g., consistently underestimating population or overestimating rates) that can be corrected with awareness. Without feedback, estimation skill plateaus at the level of intuition; with feedback, it improves measurably toward expert calibration.
How to do it
- 1Keep a log of estimates made and the actual values when they become available.
- 2Identify systematic directional biases: are your estimates consistently high or low on certain types of quantities?
- 3Apply a correction factor to domains where you have identified a persistent bias.
- 4Review the log quarterly and note where calibration has improved and where it has not.
What the evidence says
ObservationalForecasting training research consistently shows that feedback on prediction accuracy, combined with structured reflection, produces substantially better-calibrated estimators. Superforecasters’ advantage is significantly explained by their practice of reviewing and updating based on prediction outcomes.
Honest caveat: Calibration improvement requires enough predictions in a domain for the feedback to be statistically meaningful; in low-frequency domains, calibration gains are slow.
- — Tetlock & Gardner (2015), Superforecasting — tracking records as the engine of calibration improvement
Common mistake
Checking an estimate against reality once and concluding you are well calibrated — calibration is a statistical property visible only across many predictions.
IX Coach builds a running record of your estimates and their actuals, surfacing your systematic biases so you can apply corrections prospectively rather than retroactively.
Practice this with IX Coach →