PsychologyOrbit
Sign in Join

#3 ·

Spot the flaw #3: A new phonics programme in first grade

A past puzzle, with the answer. Try today's puzzle.

What is the flaw in this study?

Confound: the teachers differed. The programme classes were taught by experienced volunteers and the comparison classes by newer teachers who had not volunteered. Teaching experience and enthusiasm both affect reading progress, so the 11-point gap could come from the teachers rather than the…

A fictional study written for practice (Spot the flaw #3).

Fictional study Educational · Texas, US

The study

A school district in Texas trialled a new phonics programme in first grade. Twelve teachers volunteered to teach it; they had an average of 14 years in the classroom. Twelve other teachers, with an average of four years' experience, carried on with the usual reading curriculum. In all, 560 children took part, and the two groups had similar reading scores at the start of the year. At the end of the year children in the programme classes scored 11 points higher on the district's standard reading test, a statistically significant difference. The district's report concludes that the programme produces better readers and recommends adopting it in every elementary school.

What is the flaw in this study?

  • A Ceiling effect
  • B Confound: the teachers differed The flaw
  • C Base-rate neglect
  • D Regression to the mean
Confound: the teachers differed. The programme classes were taught by experienced volunteers and the comparison classes by newer teachers who had not volunteered. Teaching experience and enthusiasm both affect reading progress, so the 11-point gap could come from the teachers rather than the programme. The two explanations are completely tangled together. A fair test would randomly assign teachers or schools to the programme, or train the same mix of teachers in both approaches. Regression to the mean is not the problem here: classes were not selected for extreme scores, and the groups started out similar.

The concept: Confounding variables in group comparisons

This study is fictional, written for practice. No real study, institution or researcher is described.