Knowledge base Updated 2 November 2026 1 views

Replication Crisis

The finding, from the 2010s onward, that a substantial share of published psychological results do not replicate when repeated with adequate power and preregistered methods, and the reform movement that followed.

Definition

A replication attempt repeats a study's method with a new sample to see whether the result appears again. Direct replications keep the procedure as close as possible; conceptual replications test the same hypothesis by another method.

A failure to replicate does not prove the original was false. It shifts the evidence. The crisis is not that individual studies failed, but that failure rates were high, systematic, and concentrated in exactly the kinds of findings that had been most widely publicised.

History

Several events converged around 2011:

Bem published evidence for precognition in a top journal, using methods that were entirely standard. That was the point: if standard methods could produce that, standard methods were broken.
Simmons, Nelson and Simonsohn's false-positive psychology paper showed how ordinary flexibility in analysis — undisclosed choices about exclusions, conditions, covariates, when to stop collecting — could produce a significant result for almost any hypothesis, and demonstrated it by proving that listening to a song made people younger.
The Stapel fraud case broke.
The Open Science Collaboration's Reproducibility Project, published in 2015, attempted 100 replications and found that 36 percent produced significant results in the same direction, against 97 percent in the originals, with effect sizes roughly half the size.

Later efforts — Many Labs, the Social Sciences Replication Project — found similar patterns, with social psychology faring worse than cognitive psychology.

Theory

The causes are structural rather than individual dishonesty.

Publication bias toward significant, novel and surprising results.
Researcher degrees of freedom exploited without disclosure, usually without intent to deceive.
Chronically low statistical power, which both misses real effects and, combined with selective publication, inflates the effects that do get published.
Incentives that reward publication volume and novelty rather than accuracy, and that historically gave no credit for replication.
HARKing: presenting a hypothesis generated after seeing the results as if it had been specified in advance.

Research

Reforms with demonstrated effect: preregistration, Registered Reports where peer review happens before results exist, mandatory data and materials sharing, larger and multi-site samples, and the shift toward estimating effect sizes with intervals rather than testing for significance.

Registered Reports show a markedly higher rate of null results than conventionally published papers, which is the clearest available evidence that the old pipeline was filtering on outcome.

The crisis is not confined to psychology. Comparable problems have been documented in cancer biology, economics, and medicine, and psychology's visibility owes much to the fact that it investigated itself first and most publicly.

Applications

How to read a psychology paper: check whether it was preregistered and whether the analysis matches the registration; check the sample size against the effect size claimed; prefer meta-analyses that correct for publication bias; treat a single surprising study as a hypothesis, not a finding.

For practitioners, the implication is to weight interventions by the strength of replicated evidence rather than by how well-known the underlying theory is.

Criticism

Crisis may be the wrong frame. Some argue this is science working, and that the correction is evidence of health rather than sickness.
Replication is not always straightforward. Effects that depend on context can fail to replicate for reasons other than being false, and distinguishing this from special pleading is genuinely hard.
The 36 percent figure is quoted loosely. It depends on the criterion used; other criteria in the same paper give different numbers.
Reform has costs. Preregistration can discourage legitimate exploratory work if exploration is treated as illegitimate rather than simply labelled.
The burden has fallen unevenly, with early-career researchers bearing more of the cost of reform than the senior figures whose work it questioned.

References

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology. Psychological Science, 22(11), 1359-1366.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
Klein, R. A., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490.
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2).

Written for PsychologyOrbit and reviewed before publishing. It is a summary for professionals and students, not clinical guidance, and it is not a substitute for the primary sources listed above.