Knowledge base Updated 2 November 2026 1 views

P-Value

The probability of obtaining data at least as extreme as those observed, if the null hypothesis and every other assumption of the statistical model were true. It is not the probability that the hypothesis is false.

Definition

Formally, p is the probability, computed under an assumed statistical model including the null hypothesis, of a test statistic at least as extreme as the one obtained.

What it is not, in order of how often the error appears:

Not the probability the null hypothesis is true.
Not the probability the result occurred by chance.
Not the probability the finding will replicate.
Not a measure of effect size or of importance.
Not a threshold above which there is no effect. p > .05 does not mean no difference.

A p-value is a statement about data given a hypothesis, never about a hypothesis given data.

History

Fisher introduced the p-value as an informal measure of evidence against a null, to be weighed alongside everything else known.

Neyman and Pearson developed a separate decision framework with fixed error rates, alpha and beta, intended for repeated decisions rather than for evidence in a single study.

What psychology adopted is a hybrid of the two that neither camp endorsed: Fisher's p reported as evidence, compared against a Neyman-Pearson fixed threshold, with the error-rate interpretation of one and the evidential interpretation of the other. Most of the confusion in this area descends directly from that merge.

The .05 threshold has no principled basis. Fisher chose it as convenient.

Theory

Two properties make p-values easy to misuse.

They depend on the sampling plan. The same data collected under a fixed-n design and under an optional-stopping design give different p-values, because what counts as more extreme depends on what else might have happened. This is why peeking at data and continuing to collect inflates false positives.

They are very noisy. In a study with typical power, a p-value from one sample tells you remarkably little about the p-value you would get from another. Treating .04 and .06 as meaningfully different is not defensible.

Research

Misinterpretation is near-universal and documented among students, researchers and statistics instructors alike. Haller and Krauss found that a majority of methodology instructors endorsed at least one false statement about p-values.

The American Statistical Association's 2016 statement set out six principles, including that p-values do not measure the probability that the hypothesis is true and that scientific conclusions should not be based only on whether p passes a threshold. A 2019 follow-up went further and argued for abandoning statistical significance as a category.

Proposals to lower the threshold to .005 and proposals to abandon thresholds entirely have both been argued at length, and neither has settled.

Applications

Reporting practice that follows from all of this: report the exact p-value, always alongside an effect size and a confidence interval; describe results in terms of estimated magnitude and precision; do not describe a non-significant result as showing no effect; and specify the analysis in advance so the sampling plan is known.

For equivalence claims, use a test designed for them, such as two one-sided tests. A non-significant result is not evidence of equivalence.

Criticism

The hybrid framework is incoherent and is nonetheless what almost everyone is taught.
Dichotomising at .05 discards information and creates a publication filter.
P-values answer a question few people are asking. The question of interest is usually about the size of an effect, or about the probability of a hypothesis, and p addresses neither.
Alternatives have their own problems. Bayes factors depend on prior specification; estimation with intervals is better but is still misread as significance testing by another route.

References

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values. The American Statistician, 70(2), 129-133.
Greenland, S., et al. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31, 337-350.
Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers? Methods of Psychological Research Online, 7(1), 1-20.
Cumming, G. (2008). Replication and p intervals. Perspectives on Psychological Science, 3(4), 286-300.

Written for PsychologyOrbit and reviewed before publishing. It is a summary for professionals and students, not clinical guidance, and it is not a substitute for the primary sources listed above.