Reliability

Definition

Distinct forms, which are not interchangeable and are frequently treated as though they were:

• Internal consistency: do items covary. Estimated by coefficient alpha, omega, or split-half.
• Test-retest: does the score reproduce over time, for a trait expected to be stable.
• Inter-rater: do independent raters agree. Cohen's kappa, or the intraclass correlation.
• Parallel forms: do two versions give the same result.

In classical test theory, a score is a true score plus error, and reliability is the proportion of observed variance attributable to true score.

History

Spearman set out the classical model and the correction for attenuation in the early twentieth century. Cronbach's 1951 paper on alpha is one of the most cited papers in all of science.

Cronbach himself later argued that alpha had been given far more weight than it deserved and that reporting it as a single index of quality was a mistake.

Theory

What alpha is not is the practical heart of this entry.

• Alpha is not a measure of unidimensionality. A two-factor scale can have high alpha.
• Alpha increases with the number of items regardless of quality, so a long bad scale can look good.
• Alpha assumes tau-equivalence — equal item loadings — which is rarely true, and when it is violated alpha underestimates reliability. McDonald's omega does not require this assumption and is the better default.
• Alpha is a property of scores in a sample, not of an instrument. Reporting the alpha from the manual instead of computing it in the current sample is wrong and is extremely common.

Reliability limits validity: the correlation between two measures cannot exceed the geometric mean of their reliabilities. This is why low-reliability measures produce attenuated correlations that then get interpreted as small effects.

Research

Flake and Fried's audit of measurement practice in social and personality psychology found that a large share of papers used ad hoc scales, and that reliability evidence was often absent, borrowed, or reported without detail on what was computed.

Test-retest reliability of many experimental paradigms is poor. Hedge and colleagues named the paradox: tasks designed to produce robust group-level effects do so by minimising between-person variance, which is exactly what makes them unreliable as individual-difference measures. The Stroop and flanker tasks are the standard examples. Much of the literature correlating experimental task scores with questionnaires is built on this.

Applications

Scale development and evaluation; deciding whether an instrument can support individual decisions, which requires much higher reliability than group comparison does; and interpreting a change score, which needs the reliable change index rather than a raw difference.

Diagnostic and high-stakes use conventionally requires reliability above .90; .70 is adequate only for group-level research.

Criticism

• Alpha is reported reflexively and understood rarely.
• Reliability is treated as a property of the instrument rather than of scores in a population, which matters whenever an instrument is used in a new group.
• Internal consistency crowds out test-retest, even where the construct is defined as stable.
• High reliability is not evidence of validity and is regularly presented as though it were.
• The reliability paradox invalidates a substantial body of individual-difference work using cognitive tasks.

References

• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.
• McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412-433.
• Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices. Advances in Methods and Practices in Psychological Science, 3(4), 456-465.
• Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox. Behavior Research Methods, 50, 1166-1186.