Meta-analysis

From PsychWiki - A Collaborative Psychology Wiki

(Difference between revisions)
Jump to: navigation, search
(Second, choose which effect size index to calculate)
Leonbax (Talk | contribs)
m (Third, choose your statistical software)
 
(88 intermediate revisions not shown)
Line 1: Line 1:
► Have you ever wanted to learn about meta-analyses or conduct a meta-analysis but didn't know where to start?  This webpage is devoted to providing you expert opinion on what you need to know to start your own meta-analysis.
► Have you ever wanted to learn about meta-analyses or conduct a meta-analysis but didn't know where to start?  This webpage is devoted to providing you expert opinion on what you need to know to start your own meta-analysis.
-
► With the thousands of meta-analyses conducted in all areas of psychology over the past few decades, there has been an ever-increasing number of articles, books, and software programs devoted to how to conduct meta-analyses. Below, you can find out which of the many sources of information are the most useful and why -- so that the user has an easy-to-use starting place for learning everything about meta-analyses.
+
► With the thousands of meta-analyses conducted in all areas of psychology over the past few decades, there has been an ever-increasing number of articles, books, and software programs devoted to how to conduct a meta-analysis. Below, you can find out which of the many sources of information are the most useful and why -- so that you have an easy-to-use starting place for learning everything about meta-analytic reviews.
Line 12: Line 12:
==Where should I start?==
==Where should I start?==
 +
Conducting a meta-analysis is easier than it appears. While there are many, many books that describe all the intricacies of conducting a meta-analysis, try not to lose sight of the fact that a meta-analysis is essentially a straightforward process of collecting a group of studies that focus on a shared topic, and then entering statistical information into software designed to conduct meta-analysis (see [[#Third, choose your statistical software | choose your statistical software]]). The software tells you the average effect sizes from your group of studies, and also analyzes moderating variables, if that is something you are interested in examining.
<TABLE BORDER="5" style="border-top:5px solid lightgreen; border-right:5px solid lightgreen; border-bottom:5px solid yellow; border-left:5px solid yellow" cellspacing="0" cellpadding="5">
<TABLE BORDER="5" style="border-top:5px solid lightgreen; border-right:5px solid lightgreen; border-bottom:5px solid yellow; border-left:5px solid yellow" cellspacing="0" cellpadding="5">
Line 26: Line 27:
<li>For more <u>in-depth</u> discussion and explanations...</li>
<li>For more <u>in-depth</u> discussion and explanations...</li>
-
*start first with '''[[Rosenthal & DiMatteo, 2001]]''' (an Annual Review article) which provides a concise overview of everything you need to know, including the history, advantages, criticisms, and basic steps involved in a meta-analysis.
+
*start first with '''[[Rosenthal & DiMatteo, 2001]]''' (an Annual Review article) which provides a concise overview of everything you need to know, including the history, advantages, criticisms, and basic steps involved in a meta-analysis. (see the Annual Review website [http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.52.1.59 here])
-
* then try '''[[Johnson & Eagly, 2000]]''' (a chapter in Handbook of Research Methods in Social and Personality Psy) and '''[[Lipsey & Wilson, 2001]]''' (a book called Practical Meta-Analysis) for more detailed explanation of each stage in the meta-analysis process.  
+
* then try '''[[Johnson & Eagly, 2000]]''' (a chapter in the Handbook of Research Methods in Social and Personality Psychology which can be downloaded [http://digitalcommons.uconn.edu/chip_docs/13/ here]) and '''[[Lipsey & Wilson, 2001]]''' (a book called Practical Meta-Analysis) for more detailed explanation of each stage in the meta-analysis process.  
*then for even more in-depth descriptions see '''[[Cooper & Hedges, 1994]]''' (Handbook of Research Synthesis) which provides a separate chapter on every step involved in designing, analyzing, and writing-up a meta-analysis.  
*then for even more in-depth descriptions see '''[[Cooper & Hedges, 1994]]''' (Handbook of Research Synthesis) which provides a separate chapter on every step involved in designing, analyzing, and writing-up a meta-analysis.  
Line 38: Line 39:
<li>For the <u>basics</u>, see below were we lay out:</li>
<li>For the <u>basics</u>, see below were we lay out:</li>
*how to [[#First, choose which statistical approach suits your needs | choose a statistical approach]]  
*how to [[#First, choose which statistical approach suits your needs | choose a statistical approach]]  
-
*how to [[#Second, choose which effect size to calculate | choose which effect size index to calculate]]
+
*how to [[#Second, choose which effect size index to calculate | choose which effect size index to calculate]]
*how to [[#Third, choose your statistical software | choose your statistical software]]
*how to [[#Third, choose your statistical software | choose your statistical software]]
Line 61: Line 62:
A meta-analysis statistically combines the results of several studies that address a shared research hypotheses.
A meta-analysis statistically combines the results of several studies that address a shared research hypotheses.
-
Just as individual studies summarize data collected from many participants in order to answer a specific research question (e.g., each participant is a separate data-point in the analysis), a meta-analysis summarizes data from individual studies that concern a specific research question (e.g., each study is a separate data-point in the analysis).
+
Just as individual studies summarize data collected from many participants in order to answer a specific research question (i.e., each participant is a separate data-point in the analysis), a meta-analysis summarizes data from individual studies that concern a specific research question (i.e., each study is a separate data-point in the analysis).
===Three Basic Questions===
===Three Basic Questions===
:A meta-analysis answers three general questions:
:A meta-analysis answers three general questions:
-
#<u>Central tendency</u> – The central purpose of a meta-analysis is to test the relationship between two variables such that X causes Y. Central tendency refers to identifying whether X affects Y via statistically summarizing signficance levels, effect sizes, and/or confidence intervals. You are trying to answer whether X affects Y, is the effect significant, and how strong is that effect?
+
#<u>Central tendency</u> – The central purpose of a meta analysis is to test the relationship between two variables such that X affects Y. Central tendency refers to identifying whether X affects Y via statistically summarizing signficance levels, effect sizes, and/or confidence intervals. You are trying to answer whether X affects Y, is the effect significant, and how strong is that effect?
-
#<u>Variability</u> – There is always going to be some degree of variation between the outcomes of the individual studies that compose the meta-analysis. The question is whether the degree of variablity is signficantly different than what we would expect by chance alone. If so, then its called heterogeneity.
+
#<u>Variability</u> – There is always going to be some degree of variation between the outcomes of the individual studies that compose the meta-analysis. The question is whether the degree of variability is signficantly different than what we would expect by chance alone. If so, then its called heterogeneity. (for more info on heterogeneity, see [http://ssrc.tums.ac.ir/SystematicReview/Heterogeneity.asp here] and [http://www.statsdirect.com/help/meta_analysis/heterogeneity_in_meta_analysis.htm here] and [http://www.cochrane-net.org/openlearning/HTML/mod13.htm here])
#<u>Prediction</u> – If there is heterogeneity (variability), then we look for moderating variables that explain the variability. In other words, does the effect of X on Y differ with moderator variables?
#<u>Prediction</u> – If there is heterogeneity (variability), then we look for moderating variables that explain the variability. In other words, does the effect of X on Y differ with moderator variables?
Line 73: Line 74:
===Five Basic Steps===
===Five Basic Steps===
:There are generally five separate steps in conducting a meta-analysis:
:There are generally five separate steps in conducting a meta-analysis:
-
#<u>Define your Hypothesis</u> – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for a thorough examination of this step.
+
#<u>Define your hypothesis</u> – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for a thorough examination of this step.
-
#<u>Locate the Studies</u> – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (e.g., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see [[Johnson & Eagly, 2000]] (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
+
#<u>Locate the studies</u> – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (i.e., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see [[Johnson & Eagly, 2000]] (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
#<u>Input data</u> – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about [[#Third, choose your statistical software | choosing your statistical software]].  
#<u>Input data</u> – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about [[#Third, choose your statistical software | choosing your statistical software]].  
-
#<u>Cacluate Effect Sizes</u> – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like sample-size or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variablity (e.g., heterogeneity analysis). For more information see below about [[#Second, choose which effect size to calculate | choosing which effect size index to calculate]] and see [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for all the different statistical formulas.  
+
#<u>Cacluate effect sizes</u> – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like sample-size or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variability (e.g., heterogeneity analysis). For more information see below about [[#Second, choose which effect size index to calculate | choosing which effect size index to calculate]] and see [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for all the different statistical formulas.  
-
#<u>Analyze Variables</u> – If heterogeneity exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or weighted regression (for continuous variables) to see if the variable accounts for the variability in the effect size. Note - even if heterogeneity does not exist, some argue analyzing moderating variables is appropriate ([[Rosenthal, 1995]]).  FYI - [[Rosenthal, 1995]] is also an excellent Psychological Bulletin article on how to write a meta-analysis.
+
#<u>Analyze variables</u> – If heterogeneity exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or weighted regression (for continuous variables) to see if the variable accounts for the variability in the effect size. Note - even if heterogeneity does not exist, some argue analyzing moderating variables is appropriate ([[Rosenthal, 1995]]).  FYI - [[Rosenthal, 1995]] is also an excellent Psychological Bulletin article on how to write a meta-analysis.
 +
 
==How do I conduct a meta-analysis?==
==How do I conduct a meta-analysis?==
===First, choose which statistical approach suits your needs===
===First, choose which statistical approach suits your needs===
-
:There are generally three different statistical approaches to conduct a meta-analysis so first you need to choose which approach best fits your needs. For an excellent detailed comparison of these three approaches, see '''[[Johnson, Mullen, & Salas, 1995]]''' (Comparison of Three Major Meta-Analytic Approaches. Journal of Applied Psychology, 80, 94-106). Some basic information from that article is posted below to get you started:
+
:There are generally three different statistical approaches to conduct a meta-analysis so first you need to choose which approach best fits your needs. For an excellent detailed comparison of these three approaches, see [[Johnson, Mullen, & Salas, 1995]] and [[Schmidt and Hunter, 1999]]. Some basic information from the '''[[Johnson, Mullen, & Salas, 1995]]''' article is posted below to get you started:
#<u>Hedges & Olkin Approach</u> – see [[Hedges, 1981]]; [[Hedges, 1982]]; [[Hedges & Olkin, 1985]]
#<u>Hedges & Olkin Approach</u> – see [[Hedges, 1981]]; [[Hedges, 1982]]; [[Hedges & Olkin, 1985]]
#<u>Rosenthal & Rubin Approach</u> – see [[Rosenthal, 1991]]; [[Rosenthal & Rubin, 1978]]; [[Rosenthal & Rubin, 1988]]
#<u>Rosenthal & Rubin Approach</u> – see [[Rosenthal, 1991]]; [[Rosenthal & Rubin, 1978]]; [[Rosenthal & Rubin, 1988]]
Line 91: Line 93:
===Second, choose which effect size index to calculate===
===Second, choose which effect size index to calculate===
:The commonly used effect size indexes are "the "r" family and the "d" family" of effect sizes (see [[Rosenthal, 1994]]; [[Rosenthal and Dimatteo, 2001]]). Since "r" and "d" can be transformed into each other statistically you may wonder why it matters which metric you choose. Empirical research can take many forms (e.g., dichotomous and/or continuous IV, dichotomous and/or continuous DV, two variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best to use (see below). For complete information and statistical formulas for all effect size indexes for each form of research, see '''[[Lipsey & Wilson, 2001]]''' (Practical Meta-Anlaysis).
:The commonly used effect size indexes are "the "r" family and the "d" family" of effect sizes (see [[Rosenthal, 1994]]; [[Rosenthal and Dimatteo, 2001]]). Since "r" and "d" can be transformed into each other statistically you may wonder why it matters which metric you choose. Empirical research can take many forms (e.g., dichotomous and/or continuous IV, dichotomous and/or continuous DV, two variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best to use (see below). For complete information and statistical formulas for all effect size indexes for each form of research, see '''[[Lipsey & Wilson, 2001]]''' (Practical Meta-Anlaysis).
-
#<u>The '''''r''''' family</u> – Correlation Coefficient - The "r" family includes all types of correlation coefficients (e.g., ''r'', ''phi'', ''rho'', etc) and [[Johnson & Eagly, 2000]] suggest using ''r'' when the studies composing the meta-analysis primarily report the correlation between variables, but also see [[Rosenthal & DiMatteo, 2001]] for a discussion of the advantages of using ''r'' over ''d''.
+
#<u>The '''''r''''' family</u> – Correlation Coefficient - The "r" family includes all types of correlation coefficients (e.g., ''r'', ''phi'', ''rho'', etc) and [[Johnson & Eagly, 2000]] suggest using ''r'' when the studies composing the meta analysis primarily report the correlation between variables, but also see [[Rosenthal & DiMatteo, 2001]] for a discussion of the advantages of using ''r'' over ''d''.
#<u>The '''''d''''' family</u> – Standardized Difference - The "d" family includes Cohen's ''d'' (unweighted) and Hedges ''g'' (weighted), and [[Johnson & Eagly, 2000]] suggest using ''d'' when the studies composing the meta-analysis primarily report ANOVAs and t-tests comparisons between groups.
#<u>The '''''d''''' family</u> – Standardized Difference - The "d" family includes Cohen's ''d'' (unweighted) and Hedges ''g'' (weighted), and [[Johnson & Eagly, 2000]] suggest using ''d'' when the studies composing the meta-analysis primarily report ANOVAs and t-tests comparisons between groups.
 +
 +
(for an online effect size calculator for both "r" and "d", see the [http://www.lyonsmorris.com/lyons/metaAnalysis/index.cfm Larry C. Lyons website])
===Third, choose your statistical software===
===Third, choose your statistical software===
-
:You have two basic options -- use specialized software designed to conduct meta-analyses, or use standard statistical software such as SPSS and SAS. There are pros/cons to whichever option you use, so how do you choose? What you need are opinions/suggestions from those who have already used each type of software, which is where PsychWiki comes in. Directly below we have a quick summary of each approach, and then posted below are user opinions to help you identify which software is best for you!
+
:You have two basic options -- use specialized software designed to conduct meta-analyses, or use standard statistical software such as SPSS and SAS. There are pros/cons to whichever option you use, so how do you choose? What you need are opinions/suggestions from those who have already used each type of software, which is where PsychWiki comes in.  
-
#'''''[[#SPSS and SAS | SPSS and SAS]]''''' <u>(free)</u> – the [http://mason.gmu.edu/~dwilsonb/ma.html David B. Wilson website] provides an excel spreadsheet for calcuating effect sizes, and SPSS and SAS macros for perfoming meta-analyses after you have imported your effect sizes from the spreadsheet. These tools accompany the [[Lipsey and Wilson, 2001]] book Practical Meta-Analysis.
+
-
#'''''[[#Meta-Analysis | Meta-Analysis]]''''' <u>(free)</u> – Developed by [[Schwarzer, 1996]], it can be found on the [http://www.ralfschwarzer.de/ Ralf Schwarzer website] and each of the three meta-analytic approaches discussed above [[#First, choose which statistical approach suits your needs | here]] can be selected (e.g., Hedges/Olkin approach, Rosenthal approach, or Hunter/Schmidt/Jackson approach.
+
-
#'''''[[#META (Meta-Analysis Easy to Answer) | META (Meta-Analysis Easy to Answer)]]''''' <u>(free)</u> – Developed by David Kenny, a description of the software can be found on the David Kenny website [http://davidakenny.net/metain.htm here] and the program can be found on his website [http://davidakenny.net/meta.htm here].
+
-
#'''''[[#CMA (Comprehensive Meta-Analysis) | CMA (Comprehensive Meta-Analysis)]]''''' <u>(free demo, academic pricing)</u> – Developed by many of the experts in meta-analyses (see [http://www.meta-analysis.com/pages/about_us.html here] for a list), it includes an array of sophisticated options, and a comparison between CMA and other meta-analytic sofware can be found [http://www.meta-analysis.com/pages/comparisons.html here].
+
-
#'''''[[#DSTAT | DSTAT]]''''' – Developed by [[Johnson, 1989]] ...
+
-
#'''''[[#Advanced Basic Meta-analysis | Advaned Basic Meta-analysis]]''''' – Developed by [[Mullen, 1989]] ...
+
-
#'''''[[#Metawin | Metawin]]''''' – Developed by [[Rosenberg, Adams, & Gurevitch, 1997]] ...
+
-
#'''''[[#MetaDOS | MetaDOS]]''''' – Developed by [[Stauffer, 1996]] ...
+
-
 
+
::(this list is not exhaustive, so add any other software you think is useful)
-
 
+
#'''''[http://mason.gmu.edu/~dwilsonb/ma.html SPSS and SAS]''''' <u>(free)</u>
-
====SPSS and SAS====
+
#:The [http://mason.gmu.edu/~dwilsonb/ma.html David B. Wilson website] provides an excel spreadsheet for calcuating effect sizes, and SPSS and SAS macros for perfoming a meta-analysis after you have imported your effect sizes from the spreadsheet. These tools accompany the [[Lipsey and Wilson, 2001]] book Practical Meta-Analysis.
-
*
+
#'''''[http://www.meta-analysis-made-easy.com MIX 2.0]''''' <u>(free and commercial versions, academic discounts)</u>
-
*
+
#:MIX 2.0 - Professional Software for Meta-analysis in Excel. MIX 2.0 is an add-in for Excel 2007 and 2010 that allows you to perform professional meta-analysis in a familiar Excel environment. Details can be found on the [http://www.meta-analysis-made-easy.com MIX 2.0 website].
-
 
+
#'''''[http://www.ralfschwarzer.de/ Meta-Analysis]''''' <u>(free)</u>
-
 
+
#:Developed by [[Schwarzer, 1996]], it can be found on the [http://www.ralfschwarzer.de/ Ralf Schwarzer website] and each of the three meta-analytic approaches discussed above [[#First, choose which statistical approach suits your needs | here]] can be selected (i.e., Hedges/Olkin approach, Rosenthal approach, or Hunter/Schmidt/Jackson approach).
-
====Meta-Analysis====
+
#'''''[http://davidakenny.net/meta.htm META (Meta-Analysis Easy to Answer)]''''' <u>(free)</u>
-
*
+
#:Developed by David A. Kenny, a description of the software can be found on the David A. Kenny website [http://davidakenny.net/metain.htm here] and the program can be found on his website [http://davidakenny.net/meta.htm here].
-
*
+
#'''''[http://www.lyonsmorris.com/MetaA/index.htm Meta-Analysis Calculator ]''''' <u>(free)</u>
-
 
+
#:Developed by Larry C. Lyons as a web based meta-analysis application and companion to the meta-Anaysis Pages [http://www.lyonsmorris.com/MetaA/index.htm]. The applications converts individual study statistics to a common metric then accumulates the results using the Hunter-Schmidt procedures.[http://www.lyonsmorris.com/lyons/metaAnalysis/index.cfm].
-
====META (Meta-Analysis Easy to Answer)====
+
#'''''[http://www.meta-analysis.com/pages/about_us.html CMA (Comprehensive Meta-Analysis)]''''' <u>(free demo, academic pricing)</u>
-
*
+
#:Developed by many of the experts in meta-analyses (see [http://www.meta-analysis.com/pages/about_us.html here] for a list), it includes an array of sophisticated options, and a comparison between CMA and other meta-analytic sofware can be found [http://www.meta-analysis.com/pages/comparisons.html here].
-
*
+
#'''''[http://www.metawinsoft.com/ Metawin]''''' <u>(free demo, student discounts)</u>
-
 
+
#:Developed by [[Rosenberg, Adams, & Gurevitch, 1997]], see the [http://www.metawinsoft.com/ Metawin] homepage for more information including a description of Metawin [http://www.metawinsoft.com/whatismetawin.html here] and download a free demo [http://www.metawinsoft.com/mw2demo.html here].
-
====CMA (Comprehensive Meta-Analysis)====
+
#'''''[http://www.erlbaum.com/ME2/dirmod.asp?sid=28807ECF50FE49F0837125BE640E681F&nm=&type=eCommerce&mod=CommerceProductCatalog&mid=CD22EA0F118949C09A932248C040F650&tier=3&id=7F37C4A7E5464736B8870B8508B29220&itemid=1-56321-138-6 DSTAT]''''' <u>(free demo, price $25)</u>
-
*
+
#:Developed by [[Johnson, 1989]], see the [http://www.erlbaum.com/ME2/dirmod.asp?sid=28807ECF50FE49F0837125BE640E681F&nm=&type=eCommerce&mod=CommerceProductCatalog&mid=CD22EA0F118949C09A932248C040F650&tier=3&id=7F37C4A7E5464736B8870B8508B29220&itemid=1-56321-138-6 Lawrence Erlbaum website] for details.
-
*
+
#'''''Advanced Basic Meta-analysis'''''
-
*
+
#:Developed by [[Mullen, 1989]] ...
-
 
+
#'''''MetaDOS'''''
-
====DSTAT====
+
#:Developed by [[Stauffer, 1996]] ...
-
*
+
#'''''[http://www.r-project.org R-Project]''''' <u>(free, open source)</u>
-
*
+
#:Developed by collaboration, see the [http://www.r-project.org website] for details.  Meta package must be installed and loaded separately.
-
 
+
-
====Advanced Basic Meta-analysis====
+
-
*
+
-
*
+
-
 
+
-
====Metawin====
+
-
*
+
-
*
+
-
 
+
-
====MetaDOS====
+
-
*
+
-
*
+
==websites you may find interesting or helpful...==
==websites you may find interesting or helpful...==
Line 146: Line 130:
#For a powerpoint presentation summary of the [[Lipsey and Wilson, 2001]] book Practical Meta-analysis,  
#For a powerpoint presentation summary of the [[Lipsey and Wilson, 2001]] book Practical Meta-analysis,  
#:see the [http://mason.gmu.edu/~dwilsonb/ma.html David B. Wilson] website.
#:see the [http://mason.gmu.edu/~dwilsonb/ma.html David B. Wilson] website.
 +
#For a concise depiction of the meta-analytic process,
 +
#:see the [http://ssrc.tums.ac.ir/SystematicReview/Slide-Presentations.asp Systematic Review website] for a powerpoint presentation.
 +
#:see the [http://www.cochrane-net.org/openlearning/HTML/mod0-3.htm Cochrane Collaboration website] for an online booklet.
#For a truly engaging and informative paper on the history of meta-analyses written by the person who coined the term "meta-analysis"
#For a truly engaging and informative paper on the history of meta-analyses written by the person who coined the term "meta-analysis"
#:see [http://glass.ed.asu.edu/gene/papers/meta25.html Gene V. Glass] website.  
#:see [http://glass.ed.asu.edu/gene/papers/meta25.html Gene V. Glass] website.  
Line 153: Line 140:
#:see the [http://faculty.ucmerced.edu/wshadish/Meta-Analysis%20Software.htm William R. Shadish website] and  
#:see the [http://faculty.ucmerced.edu/wshadish/Meta-Analysis%20Software.htm William R. Shadish website] and  
#:see [http://www.prw.le.ac.uk/epidemio/personal/ajs22/meta/6pack.html this page] on the University of Leicester website.  
#:see [http://www.prw.le.ac.uk/epidemio/personal/ajs22/meta/6pack.html this page] on the University of Leicester website.  
 +
#:see [http://www.vetschools.co.uk/EpiVetNet/meta_analysis_software.htm this page] on EpiVetNet.
 +
#For a 2007 review of meta-analysis software,
 +
#:see [http://www.biomedcentral.com/1471-2288/7/40/ Bax et al., 2007].
#For a listing of articles that review/compare different meta-analytic software,  
#For a listing of articles that review/compare different meta-analytic software,  
#:see the [http://faculty.ucmerced.edu/wshadish/Meta-Analysis%20Software.htm William R. Shadish website] and  
#:see the [http://faculty.ucmerced.edu/wshadish/Meta-Analysis%20Software.htm William R. Shadish website] and  
#:see [http://www.prw.le.ac.uk/epidemio/personal/ajs22/meta/reviews.html this page] on the University of Leicester website.  
#:see [http://www.prw.le.ac.uk/epidemio/personal/ajs22/meta/reviews.html this page] on the University of Leicester website.  
-
#
+
#For an online professional development course on how to conduct a meta-analysis,
-
#
+
#:see the [http://www.statistics.com/ourcourses/meta] website.
-
 
+
#For the wikipedia webpage devoted to meta-analysis,
-
 
+
#:see this [http://en.wikipedia.org/wiki/Meta-analysis page].
-
 
+
#For a concise summary of the advantages and flaws of a meta-analysis:
-
 
+
#: see Medical Communications [http://www.evidence-based-medicine.co.uk/ebmfiles/WhatisMetaAn.pdf EBM page].
-
 
+
#For a Fail-Safe Number Calculator (and a paper describing the Fail-Safe Number issue),
-
 
+
#:see the [http://www.public.asu.edu/~mrosenb/software.html#failsafe Michael S. Rosenberg website].
-
 
+
-
 
+
Line 175: Line 163:
----
----
-
Back to [[Research_Tools |Research Tools mainpage]]
+
Back to [[Analyzing Data]] page

Latest revision as of 05:09, 16 November 2010

► Have you ever wanted to learn about meta-analyses or conduct a meta-analysis but didn't know where to start? This webpage is devoted to providing you expert opinion on what you need to know to start your own meta-analysis.

► With the thousands of meta-analyses conducted in all areas of psychology over the past few decades, there has been an ever-increasing number of articles, books, and software programs devoted to how to conduct a meta-analysis. Below, you can find out which of the many sources of information are the most useful and why -- so that you have an easy-to-use starting place for learning everything about meta-analytic reviews.



Contents



Where should I start?

Conducting a meta-analysis is easier than it appears. While there are many, many books that describe all the intricacies of conducting a meta-analysis, try not to lose sight of the fact that a meta-analysis is essentially a straightforward process of collecting a group of studies that focus on a shared topic, and then entering statistical information into software designed to conduct meta-analysis (see choose your statistical software). The software tells you the average effect sizes from your group of studies, and also analyzes moderating variables, if that is something you are interested in examining.

If you want to learn what is a meta-analysis...

  1. For the basics, see below where we lay out:

  2. For more in-depth discussion and explanations...
    • start first with (Rosenthal & DiMatteo, 2001) (an Annual Review article) which provides a concise overview of everything you need to know, including the history, advantages, criticisms, and basic steps involved in a meta-analysis. (see the Annual Review website here)
    • then try (Johnson & Eagly, 2000) (a chapter in the Handbook of Research Methods in Social and Personality Psychology which can be downloaded here) and (Lipsey & Wilson, 2001) (a book called Practical Meta-Analysis) for more detailed explanation of each stage in the meta-analysis process.
    • then for even more in-depth descriptions see (Cooper & Hedges, 1994) (Handbook of Research Synthesis) which provides a separate chapter on every step involved in designing, analyzing, and writing-up a meta-analysis.

If you want to learn how to start conducting a meta...

  1. For the basics, see below were we lay out:

  2. For more in-depth discussion and explanations...
    • start first with (Johnson, Mullen, & Salas, 1995) which provides a statistical comparision of the three major meta-analytic approaches using actual datasets, as well as the statistical formulas for each approach and the methodological differences between each approach.
    • if you want more information on each specific approach, see (Hedges & Olkin, 1985) for the Hedges/Olkin approach, see (Rosenthal, 1991) for the information on the Rosenthal/Rubin approach, or see (Hunter, Schmidt, & Jackson, 1982) for the Hunter/Schmidt/Jackson approach.
    • then for user-friendly step-by-step instructions for conducting all the different steps of a meta-anlaysis, see (Lipsey & Wilson, 2001) (Practical Meta-Analysis) which provides information on all three approaches including the statistical formulas for effect size indexes from each approach.




What is a meta-analysis?

Definition

A meta-analysis statistically combines the results of several studies that address a shared research hypotheses.

Just as individual studies summarize data collected from many participants in order to answer a specific research question (i.e., each participant is a separate data-point in the analysis), a meta-analysis summarizes data from individual studies that concern a specific research question (i.e., each study is a separate data-point in the analysis).


Three Basic Questions

A meta-analysis answers three general questions:
  1. Central tendency – The central purpose of a meta analysis is to test the relationship between two variables such that X affects Y. Central tendency refers to identifying whether X affects Y via statistically summarizing signficance levels, effect sizes, and/or confidence intervals. You are trying to answer whether X affects Y, is the effect significant, and how strong is that effect?
  2. Variability – There is always going to be some degree of variation between the outcomes of the individual studies that compose the meta-analysis. The question is whether the degree of variability is signficantly different than what we would expect by chance alone. If so, then its called heterogeneity. (for more info on heterogeneity, see here and here and here)
  3. Prediction – If there is heterogeneity (variability), then we look for moderating variables that explain the variability. In other words, does the effect of X on Y differ with moderator variables?


Five Basic Steps

There are generally five separate steps in conducting a meta-analysis:
  1. Define your hypothesis – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of (Lipsey & Wilson, 2001) (Practical Meta-Anlaysis) for a thorough examination of this step.
  2. Locate the studies – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (i.e., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see (Johnson & Eagly, 2000) (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
  3. Input data – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about choosing your statistical software.
  4. Cacluate effect sizes – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like sample-size or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variability (e.g., heterogeneity analysis). For more information see below about choosing which effect size index to calculate and see (Lipsey & Wilson, 2001) (Practical Meta-Anlaysis) for all the different statistical formulas.
  5. Analyze variables – If heterogeneity exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or weighted regression (for continuous variables) to see if the variable accounts for the variability in the effect size. Note - even if heterogeneity does not exist, some argue analyzing moderating variables is appropriate ((Rosenthal, 1995)). FYI - (Rosenthal, 1995) is also an excellent Psychological Bulletin article on how to write a meta-analysis.


How do I conduct a meta-analysis?

First, choose which statistical approach suits your needs

There are generally three different statistical approaches to conduct a meta-analysis so first you need to choose which approach best fits your needs. For an excellent detailed comparison of these three approaches, see (Johnson, Mullen, & Salas, 1995) and (Schmidt and Hunter, 1999). Some basic information from the (Johnson, Mullen, & Salas, 1995) article is posted below to get you started:
  1. Hedges & Olkin Approach – see (Hedges, 1981); (Hedges, 1982); (Hedges & Olkin, 1985)
  2. Rosenthal & Rubin Approach – see (Rosenthal, 1991); (Rosenthal & Rubin, 1978); (Rosenthal & Rubin, 1988)
  3. Hunter, Schmidt, & Jackson - see (Hunter, Schmidt, & Jackson, 1982); (Hunter & Schmidt, 1990)
Table1JMS2.gif


Second, choose which effect size index to calculate

The commonly used effect size indexes are "the "r" family and the "d" family" of effect sizes (see (Rosenthal, 1994); (Rosenthal and Dimatteo, 2001)). Since "r" and "d" can be transformed into each other statistically you may wonder why it matters which metric you choose. Empirical research can take many forms (e.g., dichotomous and/or continuous IV, dichotomous and/or continuous DV, two variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best to use (see below). For complete information and statistical formulas for all effect size indexes for each form of research, see (Lipsey & Wilson, 2001) (Practical Meta-Anlaysis).
  1. The r family – Correlation Coefficient - The "r" family includes all types of correlation coefficients (e.g., r, phi, rho, etc) and (Johnson & Eagly, 2000) suggest using r when the studies composing the meta analysis primarily report the correlation between variables, but also see (Rosenthal & DiMatteo, 2001) for a discussion of the advantages of using r over d.
  2. The d family – Standardized Difference - The "d" family includes Cohen's d (unweighted) and Hedges g (weighted), and (Johnson & Eagly, 2000) suggest using d when the studies composing the meta-analysis primarily report ANOVAs and t-tests comparisons between groups.

(for an online effect size calculator for both "r" and "d", see the Larry C. Lyons website)

Third, choose your statistical software

You have two basic options -- use specialized software designed to conduct meta-analyses, or use standard statistical software such as SPSS and SAS. There are pros/cons to whichever option you use, so how do you choose? What you need are opinions/suggestions from those who have already used each type of software, which is where PsychWiki comes in.
(this list is not exhaustive, so add any other software you think is useful)
  1. SPSS and SAS (free)
    The David B. Wilson website provides an excel spreadsheet for calcuating effect sizes, and SPSS and SAS macros for perfoming a meta-analysis after you have imported your effect sizes from the spreadsheet. These tools accompany the (Lipsey and Wilson, 2001) book Practical Meta-Analysis.
  2. MIX 2.0 (free and commercial versions, academic discounts)
    MIX 2.0 - Professional Software for Meta-analysis in Excel. MIX 2.0 is an add-in for Excel 2007 and 2010 that allows you to perform professional meta-analysis in a familiar Excel environment. Details can be found on the MIX 2.0 website.
  3. Meta-Analysis (free)
    Developed by (Schwarzer, 1996), it can be found on the Ralf Schwarzer website and each of the three meta-analytic approaches discussed above here can be selected (i.e., Hedges/Olkin approach, Rosenthal approach, or Hunter/Schmidt/Jackson approach).
  4. META (Meta-Analysis Easy to Answer) (free)
    Developed by David A. Kenny, a description of the software can be found on the David A. Kenny website here and the program can be found on his website here.
  5. Meta-Analysis Calculator (free)
    Developed by Larry C. Lyons as a web based meta-analysis application and companion to the meta-Anaysis Pages [1]. The applications converts individual study statistics to a common metric then accumulates the results using the Hunter-Schmidt procedures.[2].
  6. CMA (Comprehensive Meta-Analysis) (free demo, academic pricing)
    Developed by many of the experts in meta-analyses (see here for a list), it includes an array of sophisticated options, and a comparison between CMA and other meta-analytic sofware can be found here.
  7. Metawin (free demo, student discounts)
    Developed by (Rosenberg, Adams, & Gurevitch, 1997), see the Metawin homepage for more information including a description of Metawin here and download a free demo here.
  8. DSTAT (free demo, price $25)
    Developed by (Johnson, 1989), see the Lawrence Erlbaum website for details.
  9. Advanced Basic Meta-analysis
    Developed by (Mullen, 1989) ...
  10. MetaDOS
    Developed by (Stauffer, 1996) ...
  11. R-Project (free, open source)
    Developed by collaboration, see the website for details. Meta package must be installed and loaded separately.

websites you may find interesting or helpful...

  1. For an online slide-show of how to conduct a meta-analysis,
    see the University of Pittsburgh's Supercourse on how to conduct a meta-analysis.
  2. For a powerpoint presentation summary of the (Lipsey and Wilson, 2001) book Practical Meta-analysis,
    see the David B. Wilson website.
  3. For a concise depiction of the meta-analytic process,
    see the Systematic Review website for a powerpoint presentation.
    see the Cochrane Collaboration website for an online booklet.
  4. For a truly engaging and informative paper on the history of meta-analyses written by the person who coined the term "meta-analysis"
    see Gene V. Glass website.
  5. For a discussion of how a meta-analysis fits into the research process,
    see the CMA (Comprehensive Meta-Analysis) website.
  6. For a listing of various commercial and freely available meta-analysis software,
    see the William R. Shadish website and
    see this page on the University of Leicester website.
    see this page on EpiVetNet.
  7. For a 2007 review of meta-analysis software,
    see Bax et al., 2007.
  8. For a listing of articles that review/compare different meta-analytic software,
    see the William R. Shadish website and
    see this page on the University of Leicester website.
  9. For an online professional development course on how to conduct a meta-analysis,
    see the [3] website.
  10. For the wikipedia webpage devoted to meta-analysis,
    see this page.
  11. For a concise summary of the advantages and flaws of a meta-analysis:
    see Medical Communications EBM page.
  12. For a Fail-Safe Number Calculator (and a paper describing the Fail-Safe Number issue),
    see the Michael S. Rosenberg website.






◄ Back to Analyzing Data page

Personal tools
Namespaces
Variants
Actions
Navigation
Interaction
Toolbox