Assessing the Quality of Randomized Trials: Reliability of the Jadad Scale
Introduction
Meta-analyses of randomized clinical trials are being published with increasing frequency 1, 2, resulting in great interest in assessing the quality of randomized clinical trials included in meta-analyses 3, 4, 5, 6, 7, 8, 9, 10. Numerous scales and checklists have been suggested to evaluate the quality of randomized clinical trials [4]. However, a 5-item scale developed by Jadad et al. is the only known scale developed with standard scale development techniques [3]. Although the scale was developed and validated to assess the quality of reports of pain relief, it has been used extensively in other clinical areas as it is efficient to use.
Jadad reported that the scores were lower and more consistent if quality assessment was blinded [3]. The current study examines the impact of blinding on the quality score as well as the reliability of assessments by multiple raters, in publications from a different field than those evaluated by Jadad.
Section snippets
Methods
Articles describing 76 individual randomized trials were obtained from the International Study of Perioperative Transfusion (ISPOT) investigators who were conducting meta-analyses of technologies that reduce perioperative allogeneic blood transfusion during elective surgery 11, 12. The articles were presented to reviewers in a blinded and unblinded fashion (original references available from author). Blinding was achieved by masking the names and affiliations of the authors, the journal, date
Results
The reliability of the two pairs of raters, using the practice set of 10 articles, revealed a kappa of 1.00 for HC and FM, and a kappa of 0.51 for LRS and CH. The practice set of articles, using the Jadad scale, ranged in quality from 0 to 3 (maximum score of 5) with an average quality of 1.5. The pair of reviewers from France met to discuss the reasons for disagreement in the practice set and reached a consensus about their scoring differences. A conference call involving all four reviewers
Discussion
Our study suggests that there may be considerable interrater variability in the results of the Jadad scale as it is currently used. We did not repeat a second practice set of articles before proceeding with the study to establish that our agreement had improved after discussion of the issues. This certainly could also have contributed to the low kappa, but is how most research groups use a quality assessment scale. When using the Jadad scale, it may be important to ensure that good agreement is
Acknowledgements
This study was partially supported by the First International Fellowship of the International Society of Technology Assessment in Health Care, funded by the Private Patients Plan (PPP) Medical Trust, PPP Medical Institute, UK, which was awarded to Andreas Laupacis. Andreas Laupacis is a Scientist of the Medical Research Council of Canada. Finlay McAlister is a recipient of a Health Research Fellowship funded by the Medical Research Council of Canada.
References (13)
Bias in meta-analytic research
J Clin Epidemiol
(1992)- et al.
Assessing the quality of reports of randomized clinical trialsIs blinding necessary?
Control Clin Trials
(1996) - et al.
Assessing the quality of randomized controlled trialsan annotated bibliography of scales and checklists
Control Clin Trials
(1995) - et al.
Incorporating variations in the quality of individual randomized trials into meta-analysis
J Clin Epidemiol
(1992) - et al.
Meta-analysis of randomized controlled trials. A concern for standards
JAMA
(1995) Two cheers for meta-analysisproblems and opportunities in aggregating results of clinical trials
CMAJ
(1988)
Cited by (541)
Outpatient cervical ripening and labor induction with low-dose vaginal misoprostol reduces the interval to delivery: a systematic review and network meta-analysis
2024, American Journal of Obstetrics and GynecologyCadaveric Biomechanical Laboratory Research Can Be Quantitatively Scored for Quality With the Biomechanics Objective Basic Science Quality Assessment Tool: The BOBQAT Score
2024, Arthroscopy - Journal of Arthroscopic and Related SurgeryRobotic versus laparoscopic gastric bypass in bariatric surgery: a systematic review and meta-analysis on perioperative outcomes
2024, Surgery for Obesity and Related Diseases