PsychologyOrbit
Sign in Join

#14 ·

Spot the flaw #14: Ranking teams by engagement

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Unreliable measure. A test-retest correlation of .31 over two weeks means that most of the variation in answers is noise rather than a stable level of engagement. Averaging over about eight people per team helps a little, but team rankings will still shuffle from quarter to…

A fictional study written for practice (Spot the flaw #14).

Fictional study Work & organisational · Bengaluru, India

The study

The HR department of a software firm in Bengaluru wrote a single question to track staff engagement: "How engaged do you feel at work this week?", answered from 1 to 5. In a pilot, 400 employees answered it twice, two weeks apart, and the two sets of answers correlated at r = .31. The firm went ahead anyway. Each quarter it averages the answers within each of its 60 teams, which have about eight members each, and ranks the teams. Managers of the bottom ten teams lose their quarterly bonus. HR says the scheme gives managers a clear, objective incentive to improve engagement.

What is the flaw in this study?

  • A Simpson's paradox
  • B Survivorship bias
  • C Unreliable measure The flaw
  • D Order effects
Unreliable measure. A test-retest correlation of .31 over two weeks means that most of the variation in answers is noise rather than a stable level of engagement. Averaging over about eight people per team helps a little, but team rankings will still shuffle from quarter to quarter largely by chance, so managers can lose bonuses for random fluctuation. Nothing shows that the item measures engagement validly either. High-stakes decisions need a validated multi-item scale with demonstrated reliability, larger samples or repeated measurements, and an estimate of how stable the rankings really are.

The concept: Reliability limits what a measure can support

This study is fictional, written for practice. No real study, institution or researcher is described.