PsychologyOrbit
Sign in Join

#12 ·

Spot the flaw #12: A sleep-hygiene course

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Wrong test: paired data analysed as independent. The before and after scores come from the same 30 people, so they are paired, and each person's two scores are correlated. An independent-samples test treats them as 60 unrelated people: it ignores that pairing, counts stable differences between people as…

A fictional study written for practice (Spot the flaw #12).

Fictional study Health · Glasgow, UK

The study

A community health team in Glasgow ran a four-week sleep-hygiene course for 30 adults who reported poor sleep. Each person completed the Pittsburgh Sleep Quality Index (PSQI) before the course and again a week after it ended; all 30 completed both. The average score improved from 11.4 to 9.6, where lower is better. To test the change, the team treated the before scores and the after scores as two separate groups and ran an independent-samples t-test, t(58) = 1.62, p = .11. Their report concludes that the course made no measurable difference to sleep and that the funding should be redirected.

What is the flaw in this study?

  • A Ceiling effect
  • B Survivorship bias
  • C Base-rate neglect
  • D Wrong test: paired data analysed as independent The flaw
Wrong test: paired data analysed as independent. The before and after scores come from the same 30 people, so they are paired, and each person's two scores are correlated. An independent-samples test treats them as 60 unrelated people: it ignores that pairing, counts stable differences between people as error and uses the wrong degrees of freedom (58 instead of 29). A paired t-test, or a mixed model, is the right analysis and is usually more powerful. Even then, a single group cannot show that the course caused any change, so a comparison group would also help. Scores started mid-range, so a ceiling effect is not the problem.

The concept: Matching the test to the design

This study is fictional, written for practice. No real study, institution or researcher is described.