Methodological Concerns and Quality Appraisal of Contemporary Systematic Reviews and Meta-Analyses in Pediatric Urology Luis H. Braga,* Julia Pemberton, Jorge DeMaria and Armando J. Lorenzo From the Department of Surgery (Urology), McMaster Children’s Hospital and McMaster Pediatric Surgery Research Collaborative, McMaster University, Hamilton and Hospital for Sick Children, University of Toronto, Toronto (AJL), Ontario, Canada
Abbreviations and Acronyms AMSTAR ⫽ Assessment of Multiple Systematic Reviews MA ⫽ meta-analysis SR ⫽ systematic review Submitted for publication November 18, 2010. * Correspondence: Division of Urology, McMaster Children’s Hospital, 1200 Main St. W 4E19, L8N3Z5 Hamilton, Ontario, Canada (telephone: 905-521-2100, ext. 73333; FAX: 905-5219992; e-mail:
[email protected]; URL: http:// fhs.mcmaster.ca/surgery/).
See Editorial on page 15.
Purpose: The usefulness of systematic reviews and meta-analyses in influencing clinical practice depends on their quality. We sought to analyze the quality of published systematic reviews and meta-analyses in pediatric urology. Materials and Methods: We searched PubMed (MEDLINE) and Embase for all systematic reviews and meta-analyses published in the top 5 pediatric urology journals between January 2000 and November 2009. Two reviewers independently selected articles for full text review. Scientific methodological quality was evaluated using the Assessment of Multiple Systematic Reviews 11-item tool. Results: Of 267 initial results 220 articles were excluded because they were surveys, case reports or narrative reviews. Full text evaluation of the remaining 47 articles further excluded 32 series of exclusively adult patients, leaving 15 for final analysis. Seven articles (47%) were published in 2009 (p ⬍0.01). Only 1 review (7%) described a full search strategy and 3 (20%) allowed inclusion of non-English studies. In 8 reviews (53%) selection of studies was performed by 2 reviewers. Five systematic reviews (33%) described some form of quality assessment. Only 5 reviews (33%) described assessment of publication bias, while 8 (53%) checked for heterogeneity among studies. According to AMSTAR criteria, 7 systematic reviews (47%) were considered of less than fair methodological quality, 5 (33%) fair to good quality and 3 (20%) good quality. Conclusions: Despite a recent increase in the number of systematic reviews and meta-analyses published in pediatric urology journals, almost half of these reviews lack good scientific quality, raising concerns about their role in influencing clinical practice. Efforts should be made to improve the methodological quality of systematic reviews and meta-analyses in the pediatric urology literature. Key Words: child, evidence-based medicine, meta-analysis as topic, review literature as topic, urology
266
www.jurology.com
SYSTEMATIC reviews provide a comprehensive collection and summary of all available studies relevant to a focused research question. Such reviews allow for better appraisal of the published data compared to traditional narrative reviews, detecting limitations that can be addressed in future studies. Nevertheless, these advantages rely heavily on high methodological quality and strict adoption of methods
to minimize bias. In this regard it is accepted that a high quality systematic review follow the strategies of 1) clear statement of a research question, 2) a priori definition of inclusion and exclusion criteria, 3) broad and well-defined search strategy, 4) selection of studies in duplicate, 5) quality assessment of included studies, 6) duplicate data extraction with collection of information on all a priori defined
0022-5347/11/1861-0266/0 THE JOURNAL OF UROLOGY® © 2011 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION
Vol. 186, 266-272, July 2011 Printed in U.S.A. DOI:10.1016/j.juro.2011.03.044
AND
RESEARCH, INC.
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
267
outcomes, 7) careful analysis and presentation of results, and 8) discussion and explanation of heterogeneity.1 Failure to follow these principles adversely affects the quality and conclusions of the review, which may erroneously influence clinical practice. The number of published SRs and MAs has substantially increased in the last decade,1 exposing a growing number of readers to this type of methodology. Interestingly the overall quality of these reviews has been highly variable and a fair number demonstrate important deficiencies.2 These findings are concerning and underscore the importance of evaluating the quality of SRs in all fields. To our knowledge, a systematic evaluation of available SRs and MAs focusing on pediatric urology has not been conducted. We evaluated the methodological quality of such published work, checking whether the authors followed and reported the aforementioned strategies to minimize bias and errors. We hypothesize that despite widespread availability of guidelines and methodological recommendations,3,4 varying levels of quality will be encountered.
MATERIALS AND METHODS We comprehensively searched the literature for all SRs and MAs published in 5 high impact specialty specific journals, consisting of The Journal of Urology®, European Urology, Urology, BJU International and Journal of Pediatric Urology, between January 2000 and November 2009. PubMed (MEDLINE) and Embase were used as search engines, and the search was limited to the terms “humans,” “meta-analysis,” “systematic review,” “historical article,” “English” and “all children: 0 –18 years.” All search results were included in the title and abstract screening, and the following inclusion criteria were applied. Articles were published between January 2000 and November 2009. Articles had to include pediatric patients. An article had to be identified as either SR or MA by the author in the title or text, or by indexing. Narrative reviews, surveys, historical reviews and case reports with review of the literature were excluded. Articles involving only adult patients were excluded as well, while articles involving adults and children were included. Article screening was conducted independently by 2 authors (LHB, JP) and all discrepancies were resolved by consensus. Full text of each included article was obtained where possible. A data abstraction sheet was created and piloted, which included basic article characteristics such as journal, year, search strategy reported and study identification. Methodological quality was assessed using AMSTAR.5,6 According to AMSTAR criteria, a score of 0 or 1 was assigned depending on whether a criterion was met. Additive scores for each SR were calculated ranging from 0 to 11, with higher values reflecting better methodological quality. Again, 2 authors (LHB, JP) independently abstracted the full text data and completed the methodological assessment tool. Consensus meetings were held
Figure 1. Selection of reviews included in study
in case of discrepancies. The statistic was used as a measure of interobserver agreement, with ⫹1 indicating perfect agreement and –1 indicating absolute disagreement. Following the 10-item Overview of Quality Assessment Questionnaire developed by Oxman and Guyatt,7 we categorized the quality of the reviews. An AMSTAR rating of 4 or less was considered less than fair methodological quality, 5 to 8 was fair to good and 9 or greater was good.
RESULTS After initial review of titles and abstracts of 267 items we excluded 220 narrative reviews, historical articles, surveys and case reports. Subsequent full text evaluation of 47 articles resulted in further exclusion of 32 due to inappropriate age group (adults), leaving 15 studies suitable for final analysis (fig. 1). Agreement between abstractors was high ( ⫽ 0.87, 95% CI 0.69 to 1.05). Of the articles 47% (7) were published in 2009, compared to 7% to 15% for previous years (p ⬍0.01). A total of 12 SRs (80%) were exclusively pediatric, and 3 included pediatric and adult patients.8 –22 Only 4 reviews (27%) had a clearly and properly stated research question as part of the review objective. We were able to identify inclusion and exclusion criteria in 12 reviews (80%). Less than 30% of all SRs described a systematic literature search that
268
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
Figure 2. AMSTAR items reported in 15 included SRs
contained a detailed account of a full search strategy (7%), use of gray literature (theses/dissertations and conference abstracts/proceedings, 20%), inclusion of articles in a language other than English (27%), contact with experts (27%), and presentation of a list of included and excluded studies (7%). A total of 11 reviews (73%) exclusively described use of keywords to search for articles and 8 (53%) described use of at least 2 databases to search for publications. In 8 reviews (53%) abstractors judged eligibility in duplicate. Five reviews (33%) described some form of quality assessment of the included studies, and 6 (40%) mentioned that discrepancy was resolved by consensus. Five reviews (33%) described assessment of publication bias by funnel plot, while 8 (53%) checked for heterogeneity among studies and 10 (66%) presented some form of pooled statistics (meta-analysis). None of the SRs in the present study had an affiliation with the Cochrane collaboration. The percentage of SRs addressing/describing each AMSTAR criterion is outlined in figure 2. Using AMSTAR criteria, 7 reviews (47%) were considered as having less than fair methodological quality, 5 (33%) fair to good quality and 3 (20%) good quality. We compared the quality of the SRs and MAs in 2009 to those published between 2000 and 2008 to determine if the quality of reporting had improved after the creation and dissemination of the AMSTAR guidelines. Mean ⫾ SD AMSTAR score was 4.6 ⫾ 2.99 for the 7 SRs and MAs published in 2009, and
5.4 ⫾ 2.50 for the 8 articles published between 2000 and 2008 (p ⫽ 0.586, 95% CI ⫺3.94 to 2.33).
DISCUSSION When properly conducted, SRs and MAs are frequently ranked high in the hierarchy of evidence.23 Consequently their results can heavily influence clinical decisions and serve as the foundation for evidence-based practice guidelines. However, SRs can be subject to specific biases originating from poor quality of included articles, heterogeneity between studies, and presence of publication and outcome reporting biases.24 Ultimately the validity and applicability of any SR mainly depends not only on how the review was carried out, but also on the quality of the included studies. Specifically in urology the quality of included studies may represent an important limitation of SRs. In a recent study by Borawski et al 84% of 600 reviewed urological articles had level III or IV evidence.25 Another study by the same group revealed that of 152 randomized clinical trials published in 4 major urological journals only 4% included pediatric patients, confirming the low level of evidence in the pediatric urology literature.26 Our data indicate that with the recent increase in SRs and MAs in pediatric urology an important number of these articles appear to lack optimal scientific rigor due not only to inclusion of studies of
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
low level of evidence, but also (and mainly) to intrinsic methodological issues. Many of the evaluated SRs failed to achieve acceptable methodological quality due to deficiencies regarding the reported and presumably followed strategies to reduce bias. Specifically limitations in particular items applied to the research question, inclusion and exclusion criteria, search strategies, study selection, methodological quality of included studies, data extraction and synthesis of study results. Research Question Every SR should be based on a research question clearly stating the patient population, intervention, comparison intervention and outcomes. Without the research question a review lacks direction and is prone to subjective interpretation. Unfortunately this crucial step seems to be frequently overlooked. In the present study only 4 of the 15 SRs (27%) had a research question explicitly and properly stated.14,15,20,21 Inclusion and Exclusion Criteria To avoid inclusion criteria bias, inclusion and exclusion criteria must be clearly defined before data abstraction begins.24 In addition, if an MA is planned, strong consideration should be given to exclude case series, as it is ill advised to pool noncomparative studies statistically. If retrospective and prospective studies relevant to the topic are available, the authors may decide to exclude those with retrospective design due to the higher risk of bias and overestimation of treatment effects.27 In the present study although 80% of the SRs had a detailed description of inclusion and exclusion criteria, only 1 review (addressing urinary incontinence) was exclusively composed of randomized clinical trials.9 Two SRs included prospective and retrospective cohort studies.13,21 The remaining reviews included a mix of case series and observational studies.8,10 –12,14 –20,22 Consequently only 1 SR described outcomes in terms of relative risk.9 The other 14 reviews used odds ratios for the reported outcomes.8,10 –22 Search Strategy Validity is clearly linked to the search strategy used, an essential step to find all relevant studies available for review. Articles should be selected based on predefined criteria. For reproducibility it is also important to report each step of the followed strategy, depicting its outcome in detail. Unfortunately our evaluation found only 1 review with a fully described search strategy.14 Moreover, only 20% of reviews involved searching gray literature to find additional articles, 40% described the use of a single database to conduct the search and 27% described inclusion of articles in a language other than English,9,11,21 sug-
269
gesting a lack of breadth and potentially limited review of the literature. In addition, only a third of the reviews described the presence of symmetrical funnel plots, and thus the possibility that publication bias was encountered in many of these reviews cannot be underestimated. Selection of Studies To minimize the risk of investigator bias, selection of studies based on a priori established inclusion or exclusion criteria should be conducted in duplicate,27 with disagreements resolved by consensus or a third reviewer. This methodological aspect appeared well conducted in many of the evaluated SRs, showing duplicate selection in more than 50%.8,9,12,14,15,17,19,20 Optimally a flow diagram depicting the selection process in a stepwise fashion should be included, as recommend in the preferred reporting items for systematic reviews and metaanalyses statement.4 Consequently we used a similar flow diagram to summarize the selection process in our study (fig. 1). Methodological Quality of Included Studies A similar independent, dual reviewer strategy should be used to enhance objectivity and avoid error when ascertaining study quality. Although recommended, quality assessment of clinical trials is a matter of ongoing debate, reflected by the large number of different scales and checklists available.28 Nevertheless, it is an invaluable step, alerting reviewers and readers to the potential extent of bias and the resulting uncertainty in the final study pool. This is particularly true for observational studies, a design that predominates in the SRs evaluated. Although no standard quality assessment tool has been developed for observational studies, comprehensive scales have been described.29 Disappointingly only 5 reviews (33%) described some form of quality assessment of included studies,9,14,15,20,21 which negatively impacts the methodological quality of an important number of the analyzed SRs. Data Extraction Similar to the article selection and quality appraisal processes, this step should be performed independently and in duplicate, with reviewers using a piloted data extraction form to safeguard against abstractor bias.24 The type of outcome data extracted depends on the unit of measure (mean, relative risk, odds ratio or count data) that is applicable to the primary study designs used in the review. Synthesis of Study Results When analyzing data from an SR, it is crucial to determine whether quantitative synthesis is possible and appropriate. If so, pooling results can increase statistical power and allow for a more precise estimate of treatment effect.1 Nevertheless, quanti-
270
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
Figure 3. Methodological quality of 15 SRs according to AMSTAR criteria
tative synthesis (ie meta-analysis) may be unsuitable when studies are too heterogeneous. In the present study in 1 instance the authors correctly refrained from performing an MA, as they identified substantial heterogeneity in their analysis.9 Nevertheless, 10 of the 15 reviews (66%) described some form of pooled statistics.8,11,12,14 –22 It is noteworthy that statistical heterogeneity was tested in only 7 of these articles.11,12,14,15,19 –21 This finding is not isolated to the present study. On the contrary, evidence has demonstrated that only a modest fraction (45% to 68%) of SRs and MAs are tested for heterogeneity.1 Rather than ignoring heterogeneity, it should be explored by calculating tests of homogeneity (such as I2 statistic and Cochran chi-square test),1 and conducting a priori defined subgroup analyses. This issue is particularly important for SRs dealing with observational data, as clinical and statistical heterogeneity should be expected due to confounding and selection biases. Appraising Methodological Quality of SRs A recently developed tool, AMSTAR, has become particularly attractive for this purpose.5 AMSTAR includes items not previously considered, such as publication bias and disclosure of funding source.3 In the present study almost 50% of the SRs scored 4 or less (fig. 3), which implies low methodological
quality for these reviews. Unfortunately these results are not uncommon, and modest quality SRs are not limited to pediatric urology. MacDonald et al found an analogous average AMSTAR score when examining the quality of SRs published in adult urology.30 The discussed findings raise important concerns and highlight the need for critical assessment of submitted SRs. Editors and reviewers should implement measures to stimulate authors to follow and describe specific methodological steps important for adequately reporting systematically reviewed data. Such adherence may call for amendments to guidelines published by each specialty journal with particular instructions related to SRs and MAs. There are important limitations to our study that should be acknowledged. Although the number of SRs dealing with pediatric urology has increased, the overall number remains low. Consequently we are limited in assessing correlations between the detected methodological issues and publication characteristics (such as year of publication or specific journal). Moreover, some of the problems related to the quality of each SR may be due to the low level of evidence available on the topic. In addition, failure to capture specific methodological aspects of the reports may not necessarily imply that the review or MA was improperly conducted, as space
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
limitations could have led to amendments, abbreviating the text in the interest of meeting the word count limits of each journal. Nevertheless, our data appear to be in accordance with worrisome findings in other specialties. Despite an increasing trend in SR publications in recent years, a positive parallel change in quality improvement has not been observed.
271
CONCLUSIONS Despite the recent increase in SRs and MAs, an important number of these reviews appear to suffer from methodological flaws that may adversely impact their validity. Considering the potential impact of these publications, efforts should be introduced to improve the quality of SRs and MAs in the pediatric urology literature.
REFERENCES 1. Egger M, Davey Smith G and Altman DG: Systematic Reviews in Health Care. London: BMJ Books 2001. 2. Jadad AR, Cook DJ, Jones A et al: Methodology and reports of systematic reviews and metaanalyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998; 280: 278. 3. Moher D, Cook DJ, Eastwood S et al: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-Analyses. Lancet 1999; 354: 1896. 4. Moher D, Liberati A, Tetzlaff J et al: Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement. Ann Intern Med 2009; 151: 264. 5. Shea BJ, Grimshaw JM, Wells GA et al: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007; 7: 10. 6. Shea BJ, Hamel C, Wells GA et al: AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009; 62: 1013. 7. Oxman DA and Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 1991; 44: 1271. 8. Zhang JQ, Fielding JR and Zou KH: Etiology of spontaneous perirenal hemorrhage: a meta-analysis. J Urol 2002; 167: 1593.
11. Elder JS, Diaz M, Caldamone AA et al: Endoscopic therapy for vesicoureteral reflux: a metaanalysis. I. Reflux resolution and urinary tract infection. J Urol 2006; 175: 716. 12. Byun E and Merguerian PA: A meta-analysis of surgical practice patterns in the endoscopic management of ureteroceles. J Urol, suppl., 2006; 176: 1871. 13. Walsh TJ, Dall’Era MA, Croughan MS et al: Prepubertal orchiopexy for cryptorchidism may be associated with lower risk of testicular cancer. J Urol 2007; 178: 1440. 14. Guerra LA, Moher D, Sampson M et al: Intravesical oxybutynin for children with poorly compliant neurogenic bladder: a systematic review. J Urol 2008; 180: 1091. 15. Castagnetti M, Novara G, Beniamin F et al: Scintigraphic renal function after unilateral pyeloplasty in children: a systematic review. BJU Int 2008; 102: 862. 16. Barroso U Jr, Andrade D, Novaes H et al: Surgical treatment of varicocele in children with open and laparoscopic Palomo technique: a systematic review of the literature. J Urol 2009; 181: 2724. 17. Game X, Mouracade P, Chartier-Kastler E et al: Botulinum toxin-A (Botox) intradetrusor injections in children with neurogenic detrusor overactivity/ neurogenic overactive bladder: a systematic literature review. J Pediatr Urol 2009; 5: 156. 18. Kim C, McKay K and Docimo S: Laparoscopic nephrectomy in children: systematic review of transperitoneal and retroperitoneal approaches. Urology 2009; 73: 280.
9. Sureshkumar P, Bower W, Craig et al: Treatment of daytime urinary incontinence in children: a systematic review of randomized controlled trials. J Urol 2003; 170: 196.
19. Rocheleau C, Romitti P and Dennis L: Pesticides and hypospadias: a meta-analysis. J Pediatr Urol 2009; 5: 17.
10. Patel AK, Patterson JM and Chapple CR: Botulinum toxin injections for neurogenic and idiopathic detrusor overactivity: a critical analysis of results. Eur Urol 2006; 50: 684.
20. Faust WC, Diaz M and Pohl HG: Incidence of post-pyelonephritic renal scarring: a meta-analysis of the dimercapto-succinic acid literature. J Urol 2009; 181: 290.
21. Braga LH, Pace K, DeMaria J et al: Systematic review and meta-analysis of robotic-assisted versus conventional laparoscopic pyeloplasty for patients with ureteropelvic junction obstruction: effect on operative time, length of hospital stay, postoperative complications, and success rate. Eur Urol 2009; 56: 848. 22. Umbreit EC, Routh JC and Husmann DA: Nonoperative management of nonvascular grade IV blunt renal trauma in children: meta-analysis and systematic review. Urology 2009; 74: 579. 23. Hoppe DJ, Schemitsch EH, Morshed S et al: Hierarchy of evidence: where observational studies fit in and why we need them. J Bone Joint Surg Am, suppl., 2009; 91: 2. 24. Felson DT: Bias in meta-analytic research. J Clin Epidemiol 1992; 45: 885. 25. Borawski KM, Norris RD, Fesperman SF et al: Levels of evidence in the urological literature. J Urol 2007; 178: 1429. 26. Scales CD, Norris RD, Preminger GM et al: Evaluating the evidence: statistical methods in randomized controlled trials in the urological literature. J Urol 2008; 180: 1463. 27. Simunovic N, Sprague S and Bhandari M: Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic research. J Bone Joint Surg Am 2009; 91: 87. 28. Juni P, Witschi A, Bolch R et al: The hazards of scoring the quality of clinical trials for metaanalysis. JAMA 1999; 282: 1054. 29. Sanderson S, Tatt ID and Higgins JP: Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007; 36: 666. 30. MacDonald SL, Canfield SE, Fesperman SF et al: Assessment of the methodological quality of systematic reviews published in the urological literature from 1988 to 2008. J Urol 2010; 184: 648.
EDITORIAL COMMENT This is a timely and excellent article, and its publication will serve as a reference for any future SRs or MAs. However, it should not be expected that all
worthwhile SRs or MAs will be considered acceptable under either Center for Evidence-Based Medicine or AMSTAR criteria. It is important not to allow
272
QUALITY APPRAISAL OF SYSTEMATIC REVIEWS IN PEDIATRIC UROLOGY
the “perfect” to be the enemy of the “good.” When surveying the prior literature, one encounters few prospective randomized and controlled clinical trials in pediatric urology. Since disease entities that we deal with are relatively rare, organized trials with adequate statistical power will almost always require multi-institutional coordination. This undertaking is expensive and easier said than done. Most individual institutions or departments cannot adequately fund such endeavors. This article serves to underscore the need for increased National Institutes of Health funding of multicenter clinical research and trials in pediatric urology. For the same reason retrospective case series are often small in participant number. Fi-
nally, it is almost impossible to find studies whose entry criteria, treatment of patients and evaluation schema are identical to each other, making meta-analysis difficult. Recognizing these limitations, the authors succinctly outline many steps that all SRs and MAs should be able to incorporate at the outset— especially a clear statement of the research question, methodology of the literature search, inclusion and exclusion criteria, and means of data extraction. Saul P. Greenfield Division of Pediatric Urology Women and Children’s Hospital of Buffalo Buffalo, New York
