A consumer\'s guide to meta-analysis

METHODS ARTICLE

A Consumer’s Guide to Meta-Analvsis Michael LaValley Meta-analysis, an analysis that statistically pools the results from previous studies into a single quantitative analysis, has been described by one proponent as providing the very highest level of evidence for treatment efficacy (1).This type of analysis has an extensive literature (2-6) and has become a commonly employed tool in medical research. It is often used to evaluate collections of clinical trials (7,8), but has also been used to pool epidemiologic studies (9).The aim of this article is to provide a brief listing of the issues that should be considered in the construction of a solid meta-analysis. How these issues were addressed needs to be featured in the report from such a meta-analysis, and I will discuss what should be included in such a report. This is intended to help readers in critical evaluation of published meta-analyses, as well as provide guidelines on how to report a meta-analysis of one’s own. Brief explanations of the statistical issues involved in the combination of studies will be given, but readers who wish to perform their own meta-analyses should consult the source materials listed in the references for full explanations of the techniques, limitations, and interpretations of meta-analysis. The techniques used in meta-analysis were developed as systematic methods of combining the results of previously reported, and often contradictory, studies in an area in order to reach a synthesis. As such, metaanalysis is intended to provide a more objective and reproducible version of the traditional summary and review of a subject area by an expert in the field, with the expert’s experience and opinions replaced by imSupported in part by NIAMS grant SD-AR20613, awarded to the Boston University Multipurpose Arthritis and Musculoskeletal Diseases Center. Address correspondence to Michael LaValley, PhD, Boston University Arthritis Center, 80 East Concord Street, A203, Boston, MA 02118. Submitted for publication July 3,1996; accepted October 30,1996. 0 1997 by the American College of Rheumatology.

208

partial statistical formulae. Replacing traditional reviews, which often use non-scientific methods to identify and summarize information (lo), with a more systematic form of review is especially important in light of the overwhelming number of biomedical research publications (11).The need for an objective and reproducible form of review is shown by research suggesting that authors with greater experience in the area under review produce poorer quality reviews (12). Meta-analysis may also provide a more timely synthesis of the research in an area. One study found that expert recommendations for clinical treatment of potentially fatal myocardial infarctions in review articles and textbook chapters lagged years behind the results that would have been obtained by meta-analyses of randomized, placebo-controlled clinical trials (13). However, while meta-analyses are intended to be more objective and systematic than experts’ reviews, they are still subject to limitations. Because meta-analyses are essentially retrospective studies of previously collected data, they are vulnerable to criticism that the Statistical methods used, and the studies included, have been tailored to reach a desired conclusion. To blunt this criticism, it is important that a meta-analysis be conducted as a formal study. In particular, it is vital to make certain decisions prior to engaging in the metaanalysis and to include these decisions in a study proThe existence of such a protocol protocol (12,14,15). vides reassurance to readers that guidelines were in place prior to the study and that decisions were not made on an ad hoc basis. An important point that should be kept in mind when reading a meta-analysis, or performing one, is that the attention to detail in a meta-analytic study should be kept at the same level as would characterize good clinical research in other settings. It has been observed that meta-analysis may be more prone to sloppy and haphazard work than other investigative methods (16), and a study in the New England Journal of 0893-7524/97/$5.00

Arthritis Care and Research

Medicine (7) found that only 28% of published metaanalyses had adequately addressed basic questions about the conduct of the meta-analysis. Thus, we should require evidence of thoughtful and deliberate conduct from any meta-analytic study before accepting its conclusions.

Meta-analysis protocol The drawing up of a protocol is the first step that should be taken in the conduct of a meta-analysis. The protocol addresses issues related to the conduct of the study, and is analogous to the protocol in a clinical trial. The protocol should specify the research question of interest and outline the data collection and statistical methods that will be used to address the specific research question. Rather than specify how subjects will be recruited into a trial, it needs to specify how relevant candidate studies for the meta-analysis will be identified. Rather than specify - - inclusion and exclusion criteria for subjects, it needs to outline inclusion and exclusion criteria for the studies to be included in the meta-analysis. As in a clinical trial, it needs to specify the statistical methods that will be used on the collected data to address the research question. The existence and details of this protocol should be featured in the report of the meta-analysis to assure readers of the objectivity and quality of the study. While different studies will make different decisions in the protocol (e.g., one study may set the inclusion/exclusion criteria so that only randomized, doubleblind, placebo-controlled, clinical trials are used, while another study allows unblinded trials, and another includes observational studies), the existence of a protocol allows readers of a meta-analysis to see that the study was conducted in a rigorous manner.

Research hypothesis Meta-analyses often, but not exclusively, attempt to answer specific research hypotheses that have been the subject of several previous clinical trials or epidemiologic studies. Combination of many small studies is especially appealingif the studies were not of adequate size to detect a modest, but beneficial, effect. By pooling the studies together, a sample size is reached that gives adequate power to reach a conclusion, and a more definitive answer to the research question is obtained. An example of this is the meta-analysisof effect of beta-blocker drugs in the treatment of myocardial infarction (17). Thentyfour studies examining the effect of beta-blocker drugs on subjects with myocardial infarction were pooled together (20 were not statistically sigmficant) to find a statistically sigmficant reduction in mortality. Other types of research hypotheses may be addressed

Guide to Meta-Analysis 209

by meta-analysis. If the effect of treatment on a particular subgroup is of interest, combining many studies allows improved power to test results in the subgroup. The association of adverse events with treatment drug, which often cannot be determined in a single clinical trial due to the scarcity of adverse events, can be tested with improved power by meta-analysis of several trials (18). On a more exploratory level, meta-analysis can be used to explore the causes of differences between heterogeneous studies and suggest hypotheses for new studies. In fact, a prominent epidemiologist has argued that meta-analysis is most appropriate in the exploratory role (19). Whatever type of research hypothesis is chosen, it needs to be included in the study protocol and featured in the report of results. The hypothesis should be as specific as possible in order to guide the selection of studies, the type of analysis, and the diagnostic procedures performed.

Location

Of

relevant studies

Collection of relevant studies for a meta-analysiscorresponds to recruiting patients from a defined population for a clinical trial or epidemiologic study. At this point in the meta-analysis, the search should be as wide as possible. The inclusion/exclusion criteria will weed out inappropriate studies. In order to cast as wide a net as possible, and to be able to address publication bias, one should attempt to locate unpublished as well as published studies that can be used to address the research question. It has been shown that searches of computerized data bases for research studies may miss as many as 33% of relevant published studies (20). To surmount this problem, several strategies may be performed. Hand searches of journals that would publish relevant studies may be performed. References in studies that are located may reveal studies that have been missed. Contact with different researchers in the field of study may allow other studies to be found. A recent meta-analysis of the effect of fish oil on rheumatoid arthritis (21) found 7 studies by Medline search. An additional 3 unpublished studies were found by asking authors of studies and suppliers of fish oil. If the studies used in a meta-analysis are not representative of all studies in the area of research, then the conclusions of the meta-analysis will be biased. Because of this, if published studies are not representative of all studies, then using only published studies in a meta-analysis will give a biased result. Unfortunately, it seems clear that such a publication bias exists in practice. A recent survey of 11major journals found that an overwhelming proportion of published articles

Vol. 10, No. 3 , June 1997

210 LaValley

. *

.. -0.5

0.0

0.5

1.o

1.5

2.0

Log of Relative Risk

*

I

I

I

I

I

I

-0.5

0.0

0.5

1.o

1.5

2.0

Log of Relative Risk

Figure 1. Funnel plot for a collection of 50 hypothetical clinical trials (data are randomly generated). The horizontal axis gives the observed log of relative risk, and the vertical axis gives the number of subjects per arm from the 50 trials. The true value of the log relative risk is 0.51 in all trials.

Figure 2. Funnel plot after publication. Trials that did not reach significance were not published unless there were more than 100 subjects per arm. Publication bias has removed 18 studies. The pattern of the remaining 32 trials has the lower left corner removed from the funnel.

reach statistically significant conclusions (22). In the 3 medical journals they considered (AmericanJournal of Epidemiology, American Journal of Public Health, and the New England Journal of Medicine), 85.4% of the articles reached a statistically significantresult. For this to be representative of all research in the area, researchers would need to formulate only true hypotheses and to power all studies with 85% power. Clearly, the research studies published in these journals overrepresent significant findings. This bias toward significant results in publication will tend to lead to falsely significant results in meta-analyses that rely on published studies. The only clear way to avoid publication bias is to make a concerted effort to include unpublished studies in a meta-analysis. However, unpublished studies are often difficult to find, and searching through abstracts from relevant professional meetings and using word of mouth among colleagues is usually required. Another proposal to solve the problem of publication bias suggests setting up registries of all studies in a research area. This would allow investigators to access all studies, published or unpublished. The call for such registries of studies has recently been taken up by the Cochrane Collaboration,which is working with the National Library of Medicine to build the International Registry of Controlled Trials (23).

Because large studies are more likely to be published, regardless of the signdicance of the result, another approach to publication bias is to exclude small studies from the meta-analysis. The remaining large studies should be less affected by publication biases. However, such a strategy limits the number of studies included in the analysis, which may restrict the ability to detect a significant effect. For some research questions there are few studies to be used in the meta-analysis to begin with, so removal of additional studies may not be feasible. Tools for assessing the impact of publication bias on a meta-analysis include the funnel plot (2) and the tolerance for future null results (3). The funnel plot is a plot of the effect size in a study versus the number of subjects in the study. If this plot shows a rough, symmetric funnel shape, then the effect of publication bias on the meta-analysis should be slight. Figure 1 shows a collection of 50 studies before publication which, when plotted, exhibit the funnel shape. If the lower left portion of the funnel is missing, then small negative studies tended not to be published, and a publication bias is likely affecting the conclusion of the meta-analysis. Figure 2 shows the published studies from Figure 1. Publication bias can be seen in the figure because studies from the lower left corner have not been published. However, it is difficult to assess if publication bias is operating in a meta-analysis from the funnel

Guide to Meta-Analysis 211

Arthritis Care and Research

plot if there are only a few studies included in the meta-analysis (24). The tolerance for future null results is a measure of the strength of the effect found in the meta-analysis. It is the calculated number of nonsignificant studies that would need to be added to the meta-analysis to pull the result below the significance level. Rosenthal suggests that if the tolerance for future null results is more than 5 times the number of studies used in the metaanalysis plus 10 more, then a significant result would not be likely to be nullified by including unlocated, unpublished studies. All methods used to find published and unpublished studies should be included in the report of the metaanalysis to convince readers that very few, if any, relevant studies have been overlooked in the search. If the meta-analysis reaches a significant result, then funnel plots and the tolerance for future null reports should be included in the report to show readers that the results are not due to publication bias. If a null conclusion is reached by the meta-analysis, then there is no compelling need to assess publication biases in the report.

Inclusion and exclusion of studies Once all relevant studies have been gathered, decisions must be made as to which to include in the metaanalysis. As in a clinical trial, this is a crucial stage where the generality of the results in a large heterogeneous group needs to be weighed against the gain in precision from examining the results in a smaller homogeneous group. Should all studies be included in the meta-analysis, or only randomized, double-blind, placebo-controlled trials? Are blinded studies with poor randomization allowed?Are only studies from the last 10 years allowed? What sorts of variations in the treatment and outcome measure are permissible? Different researchers would make different decisions on what sorts of studies could be combined in the metaanalysis, so these decisions need to be made explicit in the protocol. It is useful to give tables for the characteristics(size, treatment effect, etc.) of trials that have been included and excluded from the meta-analysis in the report. These tables allow the reader to see any systematic differences between the studies that were included in the meta-analysis and the studies that were removed from consideration.

Quality scoring Not all research studies are conducted with the same amount of rigor. There may be flaws in the sample selection, administration of treatment (or assessment

of exposure), evaluation of results, or statistical analysis in any study. When we pool the results of different studies, we may not want to include studies with low quality, or we may want to include these studies, but give them less influence on the final result than studies with higher quality. To accomplish this, studies are often given a score that reflects the quality of the research. Assignment of a quality score to a study is best done by a “blind” judging of the study report. To accomplish this, only the methods section of the study report should be evaluated without knowledge of the study personnel or the results of the study. Because blind judging prevents the conclusion reached by the study from affecting the quality score (and relative weight of the study in the meta-analysis),it removes the bias induced by giving more weight to significant studies or studies reaching a conclusion favored by the investigator. Once quality scores have been assigned, studies with low quality scores may be removed from the analysis, or the scores may be used to adjust how much weight is given to each study relative to the others being pooled. There is some debate over how objective quality scores are (19),and if they should be used at all. Equality scores are used to weight the studies in a meta-analysis, it is useful to present the results of both the analysis that weights studies by quality score and an unweighted analysis. If studies with low scores are excluded from the analysis, both the analysis with low quality scores excluded and an analysis with all studies included should be given. In either case, if there are differences between the two results, this can be explored by examining the role of the quality scores in the analysis.

Statistical methods There are two main statistical methods for pooling studies into a meta-analysis. The first way is called a fixed effect (or Mantel-Haenzel) analysis. In a fixed effect meta-analysis, all studies are assumed to be estimating a common effect. Because of varying sample sizes, this common effect is estimated with varying precision by the different studies. The pooled estimate is a weighted average of the estimates from the individual studies, where each study is weighed by the inverse of the standard error of the estimate. A popular type of Exed effect analysis is called the Pet0 (after the meta-analysis proponent Richard Peto) or one-step method (17). However, this method can vield biased results when the effect size is large or iithe pooled studies are not balanced (25). These problems make the one-step estimator especially unsuited for pooling epidemiologic studies. The second common method is called a random

212 LaValley

effect meta-analysis (26). Here, studies are not assumed to be estimating the same effect. Rather, the true effect varies from study to study, according to a normal distribution. This variability between studies is due to small differences in the population being studied, in the treatment regimen, or in any subtle difference in conduct between the collected studies. This type of analysis was proposed to account for slight differences between studies, not large differences. If there are large differences between studies, then it is questionable whether they should be pooled. Each study is weighted by the inverse of the standard error of its estimate plus an estimate of the variability between studies. These two types of analysis usually yield similar results. A study of 22 published meta-analyses (8) found 3 where the two methods of analysis gave different results. In all 3 cases, the studies used in the meta-analysis had considerable variation in the size of treatment effect, and the fixed effect analysis was significant while the random effects analysis was not. Another difference between the two types of analysis is that small studies tend to get greater relative weight in a random effects analysis. This could bias the conclusions of the meta-analysis toward the null hypothesis if the small studies tend not to be significant (due to lack of power). However, proponents of random effects models hold that fixed effects models do not adequately address the variability that exists between studies. In general, the random effects analysis is preferred. A panel gathered by the National Research Council to study combining information suggested the use of random effects analysis as the default approach to metaanalysis (27). However, it should be noted that fixed effect analysis also has strong proponents (19,28).For more information on performing the two types of analysis, and the differences between them, see Petitti (6).

Assessment of variability among included studies There is a test of homogeneity of studies in a metaanalysis (26), which computes a test statistic based on the variability in size of treatment effect in the studies. The test statistic will be large if the results vary widely between the studies (heterogeneity), and small if the results are similar in the different studies (homogeneity). This test statistic is then compared to a cutoff from a chi-squared distribution with degrees of freedom equal to the number of studies minus one. A strategy that has been employed in meta-analyses is to perform the test of homogeneity to decide which type of analysis to employ. If the test fails to reject the null hypothesis of homogeneous studies, then a fixed effect analysis is performed. If the test rejects the hy-

Vol. 10,No. 3, June 1997

pothesis of homogeneity, then a random effects analysis is used. However, this strategy is flawed. The power of the test to detect deviations from homogeneity is usually small. So, failure to reject the null hypothesis does not give a strong evidence of homogeneity. This is another reason to use the random effects as the default analysis in a meta-analysis. The test of homogeneity should be conducted and reported in the metaanalysis to allow readers to judge the similarity of the studies that have been pooled.

Assessment of bias in the meta-analysis results Meta-analyses are subject to biases at many points. Selection of nonrepresentative studies will create a bias. Publication biases may not be adequately addressed if few unpublished studies are found. Studies may use different endpoints, populations, or treatments in ways that are not adequately addressed in the meta-analysis. Even the results abstracted from the studies may be extracted differently by different researchers. A catalog of possible biases in meta-analytic studies has been provided by Felson (29). To address the impact of bias in a meta-analysis, one should perform various sensitivity analyses. These analyses are done to assess how sensitive the final conclusions of the meta-analysis are to the particular methods used in the meta-analysis (15). Some types of sensitivity analysis have been mentioned earlier in the article: showing the results with and without use of quality scores to examine the impact of quality scores; running an analysis with only the larger studies to help determine the extent to which publication bias might affect the conclusions; performing both fixed and random effects analyses. More extensive types of sensitivity analysis are possible as well. Multiple readers could abstract results from the previous studies, and analyses using the results from each single reader could be compared. This would provide assurance that no bias was introduced into the study during the extraction of results from previous studies. In all types of sensitivity analysis, what is desired is that the conclusion of the meta-analysis not be changed when the methods are varied. As an example, consider the quality scoring issue. If an analysis using quality scores and one not using quality scores reach the same conclusion, then the meta-analysis conclusion is not sensitive to the assignment of quality score. What would be troubling is if the conclusion changes when the methods are varied. Then the conclusion could be driven by the quality scores rather than the true treatment effect. The scope of sensitivity analysis is quite large. What is important is that the sensitivity analyses cover the sources of bias that seem most likely to affect the study

Arthritis Care and Research

under consideration. These sensitivity analyses should be mentioned in the protocol and performed during the meta-analysis. If other sources of bias become evident during the conduct of the meta-analysis, then additional sensitivity analyses should be performed. All sensitivity analyses should be included in the report on the meta-analysis. This allows the reader to judge how dependent the conclusion is on the particular methods used.

summary The same proponent of meta-analysis who feels that it offers the highest level of evidence of treatment efficacy has also warned that “it is easy to do a metaanalysis; it is hard to do one well” (1).This paper has listed a series of issues that should be addressed in the construction of a good meta-analysis. A meta-analysis protocol needs to be prepared. A concerted effort should be made to capture all published and unpublished studies that address the research question. These studies should then be compared to the inclusion and exclusion criteria from the protocol to determine which will be used in the meta-analysis. A random effects analysis should be performed on the studies. A test of the homogeneity of studies should be done. Funnel plots and the tolerance for null results should be computed to assess publication biases. Finally, sensitivity analyses should be performed. All of these issues need to be discussed in the report of the meta-analysis so that readers can judge the validity of the conclusions.

REFERENCES 1. Olkin I: Meta-analysis: reconciling the results of independent studies. Stat Med 1 4 ~ 4 5 7 - 4 7 21995 , 2. Light RJ, Pillemer DB: Summing Up. Cambridge, MA, Harvard University Press, 1984 3. Rosenthal R Meta-Analytic Procedures for Social Research. Beverly Hills, CA, Sage publications, 1984 4 . Hedges LV, Olkin I: Statistical Methods for Meta-Analysis. San Diego, CA, Academic Press, 1985 5. Hunter JE, Schmidt FL: Methods of Meta-Analysis. Newbury Park, CA, Sage Publications, 1990 6. Petitti DB: Meta-Analysis, Decision Analysis, and Cost-

Effectiveness Analysis. New York, Oxford University Press, 1994 7. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers T: Meta-analysis of randomized controlled trials. New Engl J Med 316:450-455, 1987 8. Berlin JA, Laird NM, Sacks HS, Chalmers TC: A comparison of statistical methods for combining event rates from clinical trials. Stat Med 8:141-151, 1989 9 . Colditz GA, Burdick E, Mosteller F: Heterogeneity in

Guide to Meta-Analysis 213

meta-analysis of data from epidemiologic studies: a Commentary. Am J Epidemiol 142371-382, 1995 10. Mulrow CD: The medical review article: state of the science. Ann Intern Med 106:485-488, 1987 11. Mulrow CD: Rationale for systematic reviews. BMJ 309: 597-599, 1994 12. O m a n AD: Checklists for review articles. BMJ 309:648651, 1994 13. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers

TC: A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. JAMA 268:240-248, 1992 14. Gelber RD,Goldhirsch A: The concept of an overview

of cancer clinical trials with special emphasis on early breast cancer. J Clin Oncol 4:1696-1703, 1986 15. L‘Abbe KA, Detsky AS, O’Rourke K: Meta-analysis in clinical research. Ann Intern Med 107224-233, 1987 16. Ellenberg SS: Meta-analysis: the quantitative approach to research review. Semin Oncol 15:472-481, 1988 17. Yusuf S, Pet0 R, Lewis J, Collins R, Sleight T:Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis 27:335371, 1985 18. Koch GC, Schmid JE, Begun JM, Maier WC: Meta-ana-

lysis of drug safety data. In, Drug Safety Assessment in Clinical Trials. Edited by G Sogliero-Gilbert. New York, Marcel Dekker, 1993 19. Greenland S: Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol 140:290-296, 1994 20. Dickersin K, Hewitt P, Mutch L, Chalmers I, Chalmers

TC: Perusing the literature: comparison of MEDLINE searching with a perinatal clinical trials database. Control Clin Trials 6:306-317, 1985 21. Fortin PR, Lew RA, Liang MH, Wright EA, Beckett LA, Chalmers TC, Sperling RI. Validation of a meta-analysis: the effects of fish oil in rheumatoid arthritis. J Clin Epidemiol48:1379-1390, 1995 22. Sterling TD, Rosenbaum WL, Weinkam JJ: Publication

decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am Statistician 49:108-112, 1995 23. Bero L, Rennie D: The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA 274:1935-1938,1995 24. Greenland S: Letter to the editor. Am J Epidemiol 142: 1008-1009, 1995 25. Greenland S, Salvin A: Bias in the one-step method for pooling study results. Stat Med 9247-252, 1990 26. DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials 7:177-188, 1986 27. Gaver D, editor: Combining Information: Statistical Is-

sues and Opportunities for Research. Washington, DC, National Academy Press, 1992 28. Pet0 R Discussion to ‘why do we need systematic overviews of randomized trials?’ Stat Med 6241-244, 1987 29. Felson DT Bias in meta-analytic research. J Clin Epidemiol 45:885-892,1992