PsychologyOrbit
Sign in Join

#2 ·

Spot the flaw #2: A self-help leaflet for anxiety

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Regression to the mean. Patients were chosen because they scored at the very top on a single occasion. Any one score mixes a person's usual level with day-to-day fluctuation, so people picked for extreme scores will, on average, score closer to the mean when retested, even if…

A fictional study written for practice (Spot the flaw #2).

Fictional study Clinical · Manchester, UK

The study

Fourteen GP practices around Manchester screened 2,300 adult patients with the GAD-7 anxiety questionnaire during routine appointments. The 180 patients with the highest scores, all 15 or above, were posted a self-help leaflet on managing worry and invited to complete the GAD-7 again six weeks later. Of these, 171 returned it. Their average score fell from 17.2 at screening to 11.9 at follow-up, and a paired t-test showed the change was significant (p < .001). The project team reported that the leaflet cut anxiety by almost a third and recommended that practices across the region post it to every patient who screens high.

What is the flaw in this study?

  • A Ceiling effect
  • B Survivorship bias
  • C Regression to the mean The flaw
  • D Multiple comparisons without correction
Regression to the mean. Patients were chosen because they scored at the very top on a single occasion. Any one score mixes a person's usual level with day-to-day fluctuation, so people picked for extreme scores will, on average, score closer to the mean when retested, even if nothing is done. Part or all of the six-week fall is likely to be this statistical drift, not the leaflet. To separate the two, high scorers should be randomly allocated to the leaflet or to no leaflet, or screened twice before being selected. Only one outcome was tested, so multiple comparisons are not the issue.

The concept: Regression to the mean after extreme scores

This study is fictional, written for practice. No real study, institution or researcher is described.