PsychologyOrbit
Sign in Join

#5 ·

Spot the flaw #5: Favourite colours and personality

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Multiple comparisons without correction. Twenty separate tests were run at the .05 level. If colour had no link with personality at all, about one test in twenty would still come out "significant" by chance, and the chance of at least one false positive across 20 tests is about 64%. A single p = .04…

A fictional study written for practice (Spot the flaw #5).

Fictional study Social · Boston, US

The study

Psychologists in Boston recruited 400 adults online. Each chose a favourite colour from six options and completed a questionnaire measuring 20 personality facets, such as warmth, orderliness and curiosity. The team compared people who chose blue with everyone else on each of the 20 facets, using a separate t-test for each. Nineteen comparisons showed no difference. People who preferred blue scored higher on agreeableness (p = .04, d = 0.21). The study was described in a press release under the headline "Blue-lovers are kinder", and the authors suggest that colour preference could be a quick way to gauge someone's personality.

What is the flaw in this study?

  • A Multiple comparisons without correction The flaw
  • B Ceiling effect
  • C Attrition: only completers analysed
  • D Survivorship bias
Multiple comparisons without correction. Twenty separate tests were run at the .05 level. If colour had no link with personality at all, about one test in twenty would still come out "significant" by chance, and the chance of at least one false positive across 20 tests is about 64%. A single p = .04 among nineteen null results is exactly what chance predicts. The threshold should have been corrected (Bonferroni would give .0025), the hypothesis registered in advance, or the finding replicated in a new sample before any headline. The effect is also small, too small to read anyone's personality from.

The concept: False positives from many tests

This study is fictional, written for practice. No real study, institution or researcher is described.