Assessing the Quality of Randomized Trials: Reliability of the Jadad Scale

https://doi.org/10.1016/S0197-2456(99)00026-4Get rights and content

Abstract

An instrument was developed and validated by Jadad, et al. to assess the quality of clinical trials using studies from the pain literature. Our study determined the reliability of the Jadad scale and the effect of blinding on interrater agreement in another group of primary studies. Four raters independently assessed blinded and unblinded versions of 76 randomized trials. Interrater agreement was calculated among combinations of four raters for blinded and unblinded versions of the studies. A 4 × 2 × 2 repeated measures design was employed to evaluate the effect of blinding. The interrater agreement for the Jadad scale was poor (kappa 0.37 to 0.39), but agreement improved substantially (kappa 0.53 to 0.59) with removal of the third item (an explanation of withdrawals). Blinding did not significantly affect the Jadad scale scores. A more precise description of how to score the withdrawal item and careful conduct of a practice set of articles might improve interrater agreement. In contrast with the conclusions reached by Jadad, we were unable to demonstrate a significant effect of blinding on the quality scores. Control Clin Trials 1999;20:448–452

Introduction

Meta-analyses of randomized clinical trials are being published with increasing frequency 1, 2, resulting in great interest in assessing the quality of randomized clinical trials included in meta-analyses 3, 4, 5, 6, 7, 8, 9, 10. Numerous scales and checklists have been suggested to evaluate the quality of randomized clinical trials [4]. However, a 5-item scale developed by Jadad et al. is the only known scale developed with standard scale development techniques [3]. Although the scale was developed and validated to assess the quality of reports of pain relief, it has been used extensively in other clinical areas as it is efficient to use.

Jadad reported that the scores were lower and more consistent if quality assessment was blinded [3]. The current study examines the impact of blinding on the quality score as well as the reliability of assessments by multiple raters, in publications from a different field than those evaluated by Jadad.

Section snippets

Methods

Articles describing 76 individual randomized trials were obtained from the International Study of Perioperative Transfusion (ISPOT) investigators who were conducting meta-analyses of technologies that reduce perioperative allogeneic blood transfusion during elective surgery 11, 12. The articles were presented to reviewers in a blinded and unblinded fashion (original references available from author). Blinding was achieved by masking the names and affiliations of the authors, the journal, date

Results

The reliability of the two pairs of raters, using the practice set of 10 articles, revealed a kappa of 1.00 for HC and FM, and a kappa of 0.51 for LRS and CH. The practice set of articles, using the Jadad scale, ranged in quality from 0 to 3 (maximum score of 5) with an average quality of 1.5. The pair of reviewers from France met to discuss the reasons for disagreement in the practice set and reached a consensus about their scoring differences. A conference call involving all four reviewers

Discussion

Our study suggests that there may be considerable interrater variability in the results of the Jadad scale as it is currently used. We did not repeat a second practice set of articles before proceeding with the study to establish that our agreement had improved after discussion of the issues. This certainly could also have contributed to the low kappa, but is how most research groups use a quality assessment scale. When using the Jadad scale, it may be important to ensure that good agreement is

Acknowledgements

This study was partially supported by the First International Fellowship of the International Society of Technology Assessment in Health Care, funded by the Private Patients Plan (PPP) Medical Trust, PPP Medical Institute, UK, which was awarded to Andreas Laupacis. Andreas Laupacis is a Scientist of the Medical Research Council of Canada. Finlay McAlister is a recipient of a Health Research Fellowship funded by the Medical Research Council of Canada.

References (13)

There are more references available in the full text version of this article.

Cited by (541)

View all citing articles on Scopus
View full text