Structure the analysis before searching for evidence
Define what you’re looking for and what would count as evidence before starting your search.
Key takeaways
- What it is: Define what you’re looking for and what would count as evidence before starting your search.
- Why it works: Confirmation bias operates heavily in search: the framing of a search query, the databases chosen, the first result clicked — all are shaped by the prior hypothesis. Specifying in advance what evidence would count for and against the hypothesis, and what sources would be authoritative, reduces the search process’s susceptibility to motivated filtering. It converts an open-ended exploration into a pre-registered test.
- Evidence: Backed by observational / correlational evidence.
- Avoid: Specifying the hypothesis after starting to search, when the search results have already shaped what seems "important" — a subtle form of HARKing (hypothesizing after results are known).
Why it works
Confirmation bias operates heavily in search: the framing of a search query, the databases chosen, the first result clicked — all are shaped by the prior hypothesis. Specifying in advance what evidence would count for and against the hypothesis, and what sources would be authoritative, reduces the search process’s susceptibility to motivated filtering. It converts an open-ended exploration into a pre-registered test.
How to do it
- 1Before researching any question, write: "I will consider my hypothesis confirmed if I find X; I will consider it disconfirmed if I find Y."
- 2List the three or four sources that would be most credible — not the ones most likely to confirm.
- 3Commit to reporting the result of the structured search, not just the findings you happened to encounter.
- 4Do the search as designed, then evaluate.
What the evidence says
ObservationalPre-registration in science — committing to hypotheses and analysis plans before data collection — reliably reduces the rate of false positives, demonstrating that structure-before-search is an effective anti-bias intervention at the institutional level.
Honest caveat: Pre-registration works in scientific studies with clear outcomes; informal everyday evidence search is harder to pre-register in the same rigorous way.
- — Nosek et al. (2018), the preregistration revolution, PNAS
Common mistake
Specifying the hypothesis after starting to search, when the search results have already shaped what seems "important" — a subtle form of HARKing (hypothesizing after results are known).
IX Coach asks you to specify what would count as evidence before any research step, creating a structured inquiry rather than an open-ended search shaped by your priors.
Practice this with IX Coach →