Objective: To find if a particular quality score was better than others at validly scoring the quality of randomized controlled trials, both by examining the consistency of dividing studies into high and low quality and using a large study as a reference standard.
Study design and setting: Observational study of meta-analyses from the Cochrane Library. These had to have binary outcomes that included more than 10 studies, one or more of which randomized more than 500 people into each group.
Results: Eighteen systematic reviews, with 65 meta-analyses using binary outcomes, were included and the included trials were scored for 43 different quality scores. None of these scores was better at dividing the studies in to low and high quality, and none of the scores was better over the 65 meta-analyses in making the result closer to the reference standard.
Conclusion: None of the quality scores found appeared to measure quality validly. It is a mistake to assign meaning to the result of a quality score.