Statistical Power

Definition

Power is 1 minus beta, the probability of a Type II error. It is determined by the true effect size, the sample size, the alpha level, and the design.

The conventional target of .80 comes from Cohen and is a convention, not a principle. It accepts a one-in-five chance of missing a real effect, which is a strange thing to accept as standard.

Power is a property of a design plus an assumed effect size, not of a completed study. It is prospective by nature.

History

Cohen documented the problem in 1962, finding average power around .48 for a medium effect in the abnormal and social psychology literature.

Sedlmeier and Gigerenzer checked again in 1989 and found no improvement, and in some respects a decline. Cohen's paper had been widely cited and had changed nothing.

Power only entered routine practice after the replication crisis made the consequences unavoidable.

Theory

The consequences of low power extend well beyond missing effects, and this is the part usually left out of teaching.

• Type M (magnitude) error: in an underpowered study, any result that reaches significance must be large, so published effects are systematically overestimated. Gelman and Carlin formalised this.
• Type S (sign) error: with very low power, a significant result has a non-trivial probability of pointing in the wrong direction.
• Reduced positive predictive value: with low power and selective publication, the proportion of published significant findings that reflect real effects falls, which is the core of Ioannidis's argument.

The practical consequence is that a small, significant study is weaker evidence than a large, non-significant one — the opposite of how such studies are usually weighed.

Research

Median power in psychology has been repeatedly estimated in the range of .35 to .50 for typical effect sizes, meaning the median study is more likely to miss a real effect than to find it.

Post hoc power computed from the observed effect size is not informative. It is a deterministic function of the p-value and adds nothing; a non-significant result will always produce low observed power. Journals still ask for it.

Sensitivity analysis — what effect size could this design have detected — is the useful retrospective question, and is not the same thing.

Applications

Design: choose the smallest effect size of interest, not the effect a pilot study happened to produce, because pilot estimates are noisy and biased upward when selected on significance.

Other levers besides sample size: within-subject designs, more reliable measures, more trials per participant, and reducing error variance through control. Many studies could gain more power from better measurement than from more participants.

Sequential designs with pre-specified stopping rules allow efficient sampling without the false-positive inflation of unplanned peeking.

Criticism

• The .80 convention is arbitrary and is applied regardless of the cost of an error in either direction.
• Power calculations require the true effect size, which is unknown, so they rest on an assumption that is often optimistic.
• Published effect sizes are inflated, so using them for planning propagates the inflation.
• Post hoc power persists in journals and reviewer comments despite being uninformative.
• Power is not the only issue. A well-powered study of a badly-measured construct is precisely wrong.

References

• Cohen, J. (1962). The statistical power of abnormal-social psychological research. Journal of Abnormal and Social Psychology, 65(3), 145-153.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum.
• Button, K. S., et al. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.
• Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S and Type M errors. Perspectives on Psychological Science, 9(6), 641-651.
• Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power. The American Statistician, 55(1), 19-24.