PsychologyOrbit
Sign in Join

#13 ·

Spot the flaw #13: What a positive screen means

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Base-rate neglect. The leaflet confuses sensitivity with the chance that a positive result is true. Picture 1,000 members of the public: about 10 have the condition, and 9 of them screen positive. Of the 990 without it, 10%, about 99 people, also screen positive. So only around…

A fictional study written for practice (Spot the flaw #13).

Fictional study Clinical · Seattle, US

The study

A clinic network in Seattle developed a ten-question online screening tool for a condition that affects about 1 in 100 adults. In a validation study with 500 people who had a confirmed diagnosis and 500 who did not, the tool correctly flagged 90% of those with the condition and correctly cleared 90% of those without it. The network now offers the tool free on its website to the general public. Its information leaflet tells anyone who screens positive: "This test is 90% accurate, so a positive result means there is a 90% chance you have the condition. Please book an assessment."

What is the flaw in this study?

  • A Base-rate neglect The flaw
  • B Order effects
  • C Attrition: only completers analysed
  • D Restriction of range
Base-rate neglect. The leaflet confuses sensitivity with the chance that a positive result is true. Picture 1,000 members of the public: about 10 have the condition, and 9 of them screen positive. Of the 990 without it, 10%, about 99 people, also screen positive. So only around 9 of 108 positive results, roughly 8%, are true. When a condition is rare, false positives outnumber true ones even with a good test. The leaflet should give the positive predictive value for the population actually using the tool, and say that a positive screen calls for assessment, not that the condition is likely.

The concept: Positive predictive value and prevalence

This study is fictional, written for practice. No real study, institution or researcher is described.