How does "Maintain a scored prediction log" work?

Calibration requires a feedback loop: you state a probability, the outcome happens, and you learn whether your stated confidence matched the base rate of your predictions. Without recording and scoring, confidence remains a feeling rather than a measurable quantity. The Brier score is the standard metric: lower is better, rewarding confident correct predictions and penalizing confident wrong ones proportionally.

How does "Identify the specific domains where you are most overconfident" work?

People are not uniformly overconfident; they are typically more overconfident in domains where they have experience but limited feedback, where the domain uses jargon they understand, or where they have made public commitments. Diagnosing which domains produce your worst calibration focuses debiasing effort where it will have the largest return.

How does "Anchor your confidence level on base rates, not gut feel" work?

Gut-feel confidence is contaminated by how fluently the answer came to mind, how motivated you are for the outcome, and how recently a similar event happened. None of these is a reliable proxy for accuracy. Base rates from a defined reference class provide a hard external anchor that separates the confidence estimate from the emotional and cognitive noise that gut feel mixes in.

How does "Pre-commit to resolution criteria before making a prediction" work?

Vague predictions survive almost any outcome — "I said things might get complicated" is consistent with disaster or success. This outcome-flexibility is a form of motivated reasoning that prevents calibration feedback from ever arriving. Pre-committing to specific, verifiable resolution criteria closes the escape hatch: the prediction either resolves correctly or it doesn’t, and you can’t post-hoc reconstruct what you "really" meant.

How does "Build calibration reps with low-stakes trivia and almanac questions" work?

Calibration skill, like any skill, requires repetition with feedback. Real-world decisions resolve slowly, have high emotional stakes that distort honest scoring, and occur too infrequently for rapid iteration. Trivia and almanac questions provide high-volume, immediate-feedback practice in a domain with no emotional interference — making them an ideal training environment for the underlying metacognitive skill that transfers to high-stakes predictions.

How does "Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)" work?

Knight’s distinction between risk (quantifiable uncertainty) and uncertainty (unquantifiable) matters practically: assigning a 60% probability to an event requires a reference class or a model. When neither exists, the number is a feeling dressed as a statistic. Distinguishing the two prevents false precision from masking genuine ignorance, and prompts the honest response: "I don’t have enough information to put a calibrated number on this."

Philip Tetlock — Superforecasting

How do you train yourself to have well-calibrated confidence — neither overconfident nor underconfident?

Building the feedback loop between confidence and accuracy

Short answer

Calibration training teaches you to match your stated confidence levels to your actual accuracy: if you say "70% confident," roughly 70% of those claims should turn out to be correct. Tetlock’s forecasting research and probability training studies show calibration is trainable with practice and feedback — the evidence here is specific and replicable.

Most people are poorly calibrated in a specific direction: overconfident in domains where they feel competent, underconfident in new ones, and systematically bad at expressing rare-event probabilities. Calibration training is the deliberate practice of measuring the gap between stated confidence and actual accuracy, then closing it with feedback. It is not about being uncertain — it is about being accurately uncertain. Below are the practices that make the training work.

The practices (7)

Why it works

A 90% confidence interval should contain the true value 90% of the time. Most people’s 90% intervals are far too narrow — real values fall outside them 30–50% of the time, reflecting severe overconfidence in precision. Practicing interval estimation with immediate feedback recalibrates the internal sense of "how much I know," which transfers to more honest uncertainty expression in decisions.

How to do it

1Take a calibration quiz: for each factual question, state a range you’re 90% confident contains the true answer.
2After each batch of questions, score how often the true value was inside your range.
3If your 90% intervals contain the right answer less than 90% of the time, widen your ranges on the next round.
4Repeat weekly until your 90% intervals genuinely land 90% of the time — this is calibration.

Evidence

Observational

Multiple studies on probability training found that confidence interval exercises with feedback significantly reduce overconfidence in calibration tests, with effects that persist over weeks. This is among the more directly supported forms of debiasing.

Honest caveat: Calibration improvements in training tasks do not fully transfer to novel domains; calibration is partly domain-specific and requires feedback in each domain to be reliable.

— Lichtenstein, Fischhoff & Phillips (1982), "Calibration of subjective probabilities," in Judgment Under Uncertainty: Heuristics and Biases
— Tetlock & Gardner (2015), Superforecasting — calibration training modules

Common mistake: Widening all ranges uniformly rather than learning to discriminate — a well-calibrated person has narrow ranges when they genuinely know something and wide ones when they don’t, not uniformly wide ranges.

Go deeper on this practice →

Practice this with IX Coach

Reading about a practice changes nothing on its own. IX Coach turns these into a guided, adaptive routine — discerning where you are in real time and walking the practice with you, session after session.

Start with IX Coach →