Knowledge base Updated 2 November 2026 1 views

Publication Bias

The tendency for studies with significant or striking results to be published, and for null results to remain unpublished, which distorts the literature and inflates every estimate built from it.

Definition

Publication bias is one of several reporting biases, and the family is worth separating:

Publication bias: whole studies unpublished because of their results.
Outcome reporting bias: some measured outcomes reported and others dropped.
Time-lag bias: null results published later, so early meta-analyses are the most distorted.
Language and citation bias: results published or cited selectively by outcome.

The filter is applied by authors as well as by journals. Most unpublished null results were never submitted.

History

Rosenthal named the file drawer problem in 1979 and offered the fail-safe N, a method now considered unreliable and still occasionally used.

The most consequential demonstration in psychology is Turner and colleagues' 2008 comparison of antidepressant trials submitted to the FDA with the same trials as published: 94 percent of published trials were positive, against 51 percent of those registered with the regulator. Effect sizes in the published record were inflated by around a third.

Theory

The distortion compounds with low power. If most studies are underpowered and only significant results are published, then published effects are drawn from the upper tail of the sampling distribution, and the meta-analytic average of a biased set is a biased average.

This is why meta-analysis does not automatically fix the problem. A meta-analysis of a filtered literature is a precise estimate of the filter.

P-hacking and publication bias produce different signatures. Publication bias thins the region just above the threshold from the null side; p-hacking piles results up just below .05. The distinction underlies p-curve analysis.

Research

Detection methods, none of them decisive:

Funnel plot asymmetry and Egger's test. Low power with few studies, and asymmetry has causes other than bias, including genuine small-study effects.
Trim and fill. Widely used, and known to perform poorly under heterogeneity; it can over- or under-correct badly.
P-curve and p-uniform, which examine the distribution of significant p-values to estimate evidential value.
Selection models, which model the publication process explicitly and are the most principled option, at the cost of requiring assumptions about the selection mechanism.

Applied to psychology, these methods repeatedly reduce estimated effects, sometimes to near zero. Several well-known literatures — ego depletion is the clearest case — collapse under bias-corrected analysis.

Applications

Reading meta-analyses: check whether bias was assessed, by what method, and what the corrected estimate was. A meta-analysis without a bias assessment should be discounted, and one that used only trim and fill should be treated cautiously.

Preventing it: preregistration, Registered Reports, mandatory trial registration, and journals willing to publish informative nulls. The last is the one the field has changed least.

Criticism

Correction methods disagree with each other on the same data, and there is no consensus on which to trust.
Trim and fill is used far beyond its valid range.
Small-study effects are not always bias. Smaller studies may genuinely differ in population or in fidelity of delivery.
Assessment requires many studies, so the small meta-analyses common in psychology cannot detect bias reliably.
The incentives that cause it are intact. Registration helps at the margins; the underlying reward structure still pays for positive results.

References

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.
Turner, E. H., et al. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358(3), 252-260.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534-547.
Carter, E. C., et al. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2(2), 115-144.
Duval, S., & Tweedie, R. (2000). Trim and fill. Biometrics, 56(2), 455-463.

Written for PsychologyOrbit and reviewed before publishing. It is a summary for professionals and students, not clinical guidance, and it is not a substitute for the primary sources listed above.