Evaluating an RCT report: CI for a difference, differential drop-out, contamination
education
A company trialled a maths practice app with 600 Year 9 students from 12 schools. Within each school, students were randomly allocated to use the app (300) or do usual homework (300) for two terms. At the end, students sat a test; the outcome was whether they reached curriculum level 5. Figure 1 shows the design of the trial.
App group: 240 sat the test (60 had left the trial), 156 reached level 5. Homework group: 285 sat the test (15 had left), 157 reached level 5. Figure 2 shows what happened to the 300 students allocated to each group.
| Group | Allocated | Left the trial | Sat test | Reached level 5 |
|---|---|---|---|---|
| App | 300 | 60 | 240 | 156 |
| Homework | 300 | 15 | 285 | 157 |
The company's report says: 'The app boosted the proportion of students reaching level 5 by 10 percentage points.'
Flow diagram: 600 Year 9 students from 12 schools are randomly allocated within each school to the app group (300) or the usual homework group (300) for two terms, and then all sit a test to see whether they reach curriculum level 5.
Stacked horizontal bar chart of the 300 students allocated to each group, split into those who reached level 5, those who sat the test but did not reach level 5, and those who left the trial (app: 156, 84, 60; homework: 157, 128, 15).Facts you may need
(a)
Calculate the percentage reaching level 5 in each group among those who sat the test, and construct a confidence interval for the difference.
(b)
Use Figure 2 to explain how the different drop-out rates could produce the difference, and calculate the percentage reaching level 5 in each group if every student who left had not reached level 5.
(c)
Students were randomly allocated within the same schools. Discuss one advantage and one disadvantage of this.
(d)
Write a fairer one-sentence summary of the trial's findings. Explain why your wording is better.
Maths app trial for Year 9 students — Scholarship Statistics | Xametric
