PsychologyOrbit
Sign in Join

#6 ·

Spot the flaw #6: Group therapy for insomnia

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Underpowered sample. With nine people per group, the study had little chance of detecting even a real, moderate effect, so a non-significant result tells us very little. "Not significant" is not the same as "no effect": the therapy group improved by three points more, a…

A fictional study written for practice (Spot the flaw #6).

Fictional study Clinical · Karachi, Pakistan

The study

A sleep clinic at a hospital in Karachi randomly allocated 18 adults with chronic insomnia to either six weekly sessions of group cognitive behavioural therapy for insomnia (9 people) or a waiting list (9 people). Everyone completed the Insomnia Severity Index before and after; nobody dropped out. Scores fell by 5.1 points in the therapy group and by 2.0 points on the waiting list. The difference between the groups was not statistically significant (p = .12). The authors conclude that group therapy does not work for insomnia in Pakistani adults and advise clinics not to offer it.

What is the flaw in this study?

  • A No random allocation
  • B Survivorship bias
  • C Underpowered sample The flaw
  • D Ceiling effect
Underpowered sample. With nine people per group, the study had little chance of detecting even a real, moderate effect, so a non-significant result tells us very little. "Not significant" is not the same as "no effect": the therapy group improved by three points more, a difference many clinicians would value, and the confidence interval would be wide enough to include a large benefit. A power analysis beforehand would have shown that around 64 people per group are needed to detect a medium effect (d = 0.5) with 80% power. Allocation was random, so that is not the flaw.

The concept: Statistical power and null results

This study is fictional, written for practice. No real study, institution or researcher is described.