Metaanalysis
From PsychWiki  A Collaborative Psychology Wiki
Hi, I am your guide, Doug Stenstrom ^{Where can I go if I have a question?} 
► With the thousands of metaanalyses conducted in all areas of psychology over the past few decades, there has been an everincreasing number of articles, books, and software programs devoted to how to conduct a metaanalysis. Below, you can find out which of the many sources of information are the most useful and why  so that you have an easytouse starting place for learning everything about metaanalytic reviews.
Contents 
Where should I start?
If you want to learn what is a metaanalysis...

If you want to learn how to start conducting a meta...

What is a metaanalysis?
Definition
A metaanalysis statistically combines the results of several studies that address a shared research hypotheses.
Just as individual studies summarize data collected from many participants in order to answer a specific research question (i.e., each participant is a separate datapoint in the analysis), a metaanalysis summarizes data from individual studies that concern a specific research question (i.e., each study is a separate datapoint in the analysis).
Three Basic Questions
 A metaanalysis answers three general questions:
 Central tendency – The central purpose of a meta analysis is to test the relationship between two variables such that X affects Y. Central tendency refers to identifying whether X affects Y via statistically summarizing signficance levels, effect sizes, and/or confidence intervals. You are trying to answer whether X affects Y, is the effect significant, and how strong is that effect?
 Variability – There is always going to be some degree of variation between the outcomes of the individual studies that compose the metaanalysis. The question is whether the degree of variability is signficantly different than what we would expect by chance alone. If so, then its called heterogeneity. (for more info on heterogeneity, see here and here and here)
 Prediction – If there is heterogeneity (variability), then we look for moderating variables that explain the variability. In other words, does the effect of X on Y differ with moderator variables?
Five Basic Steps
 There are generally five separate steps in conducting a metaanalysis:
 Define your hypothesis – First you must have a welldefined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of (Lipsey & Wilson, 2001) (Practical MetaAnlaysis) for a thorough examination of this step.
 Locate the studies – A metaanalysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (i.e., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see (Johnson & Eagly, 2000) (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
 Input data – Gather empiricial findings from primary studies (e.g., pvalue, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about choosing your statistical software.
 Cacluate effect sizes – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like samplesize or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variability (e.g., heterogeneity analysis). For more information see below about choosing which effect size index to calculate and see (Lipsey & Wilson, 2001) (Practical MetaAnlaysis) for all the different statistical formulas.
 Analyze variables – If heterogeneity exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or weighted regression (for continuous variables) to see if the variable accounts for the variability in the effect size. Note  even if heterogeneity does not exist, some argue analyzing moderating variables is appropriate ((Rosenthal, 1995)). FYI  (Rosenthal, 1995) is also an excellent Psychological Bulletin article on how to write a metaanalysis.
How do I conduct a metaanalysis?
First, choose which statistical approach suits your needs
 There are generally three different statistical approaches to conduct a metaanalysis so first you need to choose which approach best fits your needs. For an excellent detailed comparison of these three approaches, see (Johnson, Mullen, & Salas, 1995) and (Schmidt and Hunter, 1999). Some basic information from the (Johnson, Mullen, & Salas, 1995) article is posted below to get you started:
 Hedges & Olkin Approach – see (Hedges, 1981); (Hedges, 1982); (Hedges & Olkin, 1985)
 Rosenthal & Rubin Approach – see (Rosenthal, 1991); (Rosenthal & Rubin, 1978); (Rosenthal & Rubin, 1988)
 Hunter, Schmidt, & Jackson  see (Hunter, Schmidt, & Jackson, 1982); (Hunter & Schmidt, 1990)
Second, choose which effect size index to calculate
 The commonly used effect size indexes are "the "r" family and the "d" family" of effect sizes (see (Rosenthal, 1994); (Rosenthal and Dimatteo, 2001)). Since "r" and "d" can be transformed into each other statistically you may wonder why it matters which metric you choose. Empirical research can take many forms (e.g., dichotomous and/or continuous IV, dichotomous and/or continuous DV, two variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best to use (see below). For complete information and statistical formulas for all effect size indexes for each form of research, see (Lipsey & Wilson, 2001) (Practical MetaAnlaysis).
 The r family – Correlation Coefficient  The "r" family includes all types of correlation coefficients (e.g., r, phi, rho, etc) and (Johnson & Eagly, 2000) suggest using r when the studies composing the meta analysis primarily report the correlation between variables, but also see (Rosenthal & DiMatteo, 2001) for a discussion of the advantages of using r over d.
 The d family – Standardized Difference  The "d" family includes Cohen's d (unweighted) and Hedges g (weighted), and (Johnson & Eagly, 2000) suggest using d when the studies composing the metaanalysis primarily report ANOVAs and ttests comparisons between groups.
(for an online effect size calculator for both "r" and "d", see the Larry C. Lyons website)
Third, choose your statistical software
 You have two basic options  use specialized software designed to conduct metaanalyses, or use standard statistical software such as SPSS and SAS. There are pros/cons to whichever option you use, so how do you choose? What you need are opinions/suggestions from those who have already used each type of software, which is where PsychWiki comes in. Directly below we have a quick summary of each approach, and then posted next are user opinions to help you identify which software is best for you!
 (this list is not exhaustive, so add any other software you think is useful)
 SPSS and SAS (free) – the David B. Wilson website provides an excel spreadsheet for calcuating effect sizes, and SPSS and SAS macros for perfoming a metaanalysis after you have imported your effect sizes from the spreadsheet. These tools accompany the (Lipsey and Wilson, 2001) book Practical MetaAnalysis.
 MIX (free)  MIX  Metaanalysis with Interactive eXplanations  is an Excelbased software that can be found here.
 MetaAnalysis (free) – Developed by (Schwarzer, 1996), it can be found on the Ralf Schwarzer website and each of the three metaanalytic approaches discussed above here can be selected (i.e., Hedges/Olkin approach, Rosenthal approach, or Hunter/Schmidt/Jackson approach).
 META (MetaAnalysis Easy to Answer) (free) – Developed by David A. Kenny, a description of the software can be found on the David A. Kenny website here and the program can be found on his website here.
 MetaAnalysis Calculator (free) – Developed by Larry C. Lyons as a web based metaanalysis application and companion to the metaAnaysis Pages [1]. The applications converts individual study statistics to a common metric then accumulates the results using the HunterSchmidt procedures.[2].
 CMA (Comprehensive MetaAnalysis) (free demo, academic pricing) – Developed by many of the experts in metaanalyses (see here for a list), it includes an array of sophisticated options, and a comparison between CMA and other metaanalytic sofware can be found here.
 Metawin (free demo, student discounts) – Developed by (Rosenberg, Adams, & Gurevitch, 1997), see the Metawin homepage for more information including a description of Metawin here and download a free demo here.
 DSTAT (free demo, price $25) – Developed by (Johnson, 1989), see the Lawrence Erlbaum website for details.
 Advanced Basic Metaanalysis – Developed by (Mullen, 1989) ...
 MetaDOS – Developed by (Stauffer, 1996) ...
User Opinions
SPSS and SAS
MIX
MetaAnalysis
META
CMA
Metawin
DSTAT
Advanced Basic
MetaDOS
websites you may find interesting or helpful...
 For an online slideshow of how to conduct a metaanalysis,
 see the University of Pittsburgh's Supercourse on how to conduct a metaanalysis.
 For a powerpoint presentation summary of the (Lipsey and Wilson, 2001) book Practical Metaanalysis,
 see the David B. Wilson website.
 For a concise depiction of the metaanalytic process,
 see the Systematic Review website for a powerpoint presentation.
 see the Cochrane Collaboration website for an online booklet.
 For a truly engaging and informative paper on the history of metaanalyses written by the person who coined the term "metaanalysis"
 see Gene V. Glass website.
 For a discussion of how a metaanalysis fits into the research process,
 see the CMA (Comprehensive MetaAnalysis) website.
 For a listing of various commercial and freely available metaanalysis software,
 see the William R. Shadish website and
 see this page on the University of Leicester website.
 see this page on EpiVetNet.
 For a listing of articles that review/compare different metaanalytic software,
 see the William R. Shadish website and
 see this page on the University of Leicester website.
 For an online professional development course on how to conduct a metaanalysis,
 see the Statistics.com website.
 For the wikipedia webpage devoted to metaanalysis,
 see this page.
 For a concise summary of the advantages and flaws of a metaanalysis:
 see Medical Communications EBM page.
 For a FailSafe Number Calculator (and a paper describing the FailSafe Number issue),
 see the Michael S. Rosenberg website.
◄ Back to Research Tools mainpage