Treat forecasting accuracy as a skill that improves with practice
Take prediction errors as performance feedback, not as proof that forecasting is futile.
Key takeaways
- What it is: Take prediction errors as performance feedback, not as proof that forecasting is futile.
- Why it works: Superforecasters treat their scoring results as diagnostics, not verdicts. When they are miscalibrated in a domain, they investigate why — was the reference class wrong, was the evidence weighting off, was there a structural dependency they missed? This feedback orientation is the same mechanism that distinguishes deliberate practice from mere experience. Without it, repetition produces confidence without accuracy.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Using a single dramatic miss to conclude that forecasting is unreliable rather than investigating the specific mechanism of the miss — one error is a datum, not a refutation.
Why it works
Superforecasters treat their scoring results as diagnostics, not verdicts. When they are miscalibrated in a domain, they investigate why — was the reference class wrong, was the evidence weighting off, was there a structural dependency they missed? This feedback orientation is the same mechanism that distinguishes deliberate practice from mere experience. Without it, repetition produces confidence without accuracy.
How to do it
- 1After a prediction resolves, score it: compare your stated probability to the binary outcome.
- 2For predictions you got significantly wrong, run a brief post-mortem: what was the strongest factor you underweighted?
- 3Track error patterns: do you over-predict in certain domains, under-predict in others?
- 4Set a learning goal for calibration (not just accuracy) — "I want my 70% predictions to land 70% of the time."
What the evidence says
ObservationalTetlock’s research found that forecasters who engaged in deliberate learning from scoring — particularly those who practiced in teams with feedback — showed the most improvement over time. The growth trajectory is consistent with deliberate practice research.
Honest caveat: Calibration improvement is slow and context-dependent; large improvements were seen in tournament conditions with dense feedback. Real-world decision-making with sparse feedback produces slower gains.
- — Tetlock & Gardner (2015), Superforecasting, Chapter 10
Common mistake
Using a single dramatic miss to conclude that forecasting is unreliable rather than investigating the specific mechanism of the miss — one error is a datum, not a refutation.
IX Coach scores your historical confidence estimates against outcomes over time, generating a personal calibration report that turns each session’s data into a learning signal.
Practice this with IX Coach →