PsychologyOrbit
Sign in Join

#9 ·

Spot the flaw #9: A resilience course at work

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Attrition: only completers analysed. Six in ten people left the course, and only those who stayed were analysed. People who finish a demanding course are often the more motivated, less overloaded or already coping better, so the finishers are no longer comparable to the full waiting-list group…

A fictional study written for practice (Spot the flaw #9).

Fictional study Work & organisational · Ohio, US

The study

A manufacturing company in Ohio randomly allocated 500 staff to a 12-week resilience course or to a waiting list. Almost everyone on the waiting list completed the before and after questionnaires, and their scores changed little. On the course, 150 of the 250 people stopped attending, most of them in the first month, and did not complete the final questionnaire. The analysis compared the 100 people who finished the course with the waiting list. Among the finishers, resilience scores rose by 22% and burnout scores fell sharply. The company concluded that the course works and made it compulsory for all staff.

What is the flaw in this study?

  • A Attrition: only completers analysed The flaw
  • B Ceiling effect
  • C Base-rate neglect
  • D Simpson's paradox
Attrition: only completers analysed. Six in ten people left the course, and only those who stayed were analysed. People who finish a demanding course are often the more motivated, less overloaded or already coping better, so the finishers are no longer comparable to the full waiting-list group. The random allocation that made the groups equal has been undone. The fair comparison is an intention-to-treat analysis of all 250 people allocated to the course, with a sensible method for missing data, plus a report of why people dropped out. Making a course compulsory when most people abandoned it is a further warning sign.

The concept: Intention-to-treat and dropout bias

This study is fictional, written for practice. No real study, institution or researcher is described.