Meta-analysis

From PsychWiki - A Collaborative Psychology Wiki

(Difference between revisions)
Jump to: navigation, search
Stenstro (Talk | contribs)
Line 72: Line 72:
===Five Basic Steps===
===Five Basic Steps===
:There are generally five separate steps in conducting a meta-analysis:
:There are generally five separate steps in conducting a meta-analysis:
-
#<u>Define your Hypothesis</u> – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. (see Chapter 2 of [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for a thorough examination of this step).
+
#<u>Define your Hypothesis</u> – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of [[Lipsey & Wilson, 2001]] (Practical Meta-Anlaysis) for a thorough examination of this step.
-
#<u>Locate the Studies</u> – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (e.g., network of researchers who know about unpublished studies, conference proceedings, etc)(see [[Johnson & Eagly, 2000]] (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles).
+
#<u>Locate the Studies</u> – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (e.g., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see [[Johnson & Eagly, 2000]] (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
-
#<u>Input data</u> – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database (see below about [[#Third, choose your statistical software | choosing your statistical software]]). Not every study provides sufficient statistical information to calculate the effect size statistic.  
+
#<u>Input data</u> – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about [[#Third, choose your statistical software | choosing your statistical software]].  
-
#<u>Cacluate Effect Sizes</u> – Calculate the overall effect by converting all statistics to a common metric (see below about [[#Second, choose which effect size to calculate? | choosing which effect size to calculate]]), and  
+
#<u>Cacluate Effect Sizes</u> – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like sample-size or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variablity (e.g., homogeneity/heterogeneity analysis). For more information see below about [[#Second, choose which effect size to calculate? | choosing which effect size to calculate]] and see any meta-analysis book for all the statistical formulas.  
-
 
+
-
-- effect size, significance level or confidence intervals associated with the effect size, and variaiblity (homogeneity) of the effect.
+
#<u>Analyze Variables</u> – If heterogeneity (variability) exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or bivariate prediction (for continuous variables) to see if the variable accounts for the variability in the effect size.
#<u>Analyze Variables</u> – If heterogeneity (variability) exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or bivariate prediction (for continuous variables) to see if the variable accounts for the variability in the effect size.
-
===Two x Two types of Variables===
+
===Two (x Two) types of Variables===
 +
:In addition to variables being categorical (e.g., discrete categories) or continuous (e.g., assigned Likert scale value), ....
 +
 
There are two types of study variables: (a) objective variables -- such as type of IV or DV, ..., (b) subjective variables -- inferential judgements made by two or more judges....
There are two types of study variables: (a) objective variables -- such as type of IV or DV, ..., (b) subjective variables -- inferential judgements made by two or more judges....

Revision as of 05:30, 21 July 2006

► Have you ever wanted to learn about meta-analysis or conduct a meta-analysis but didn't know where to start? This webpage is devoted to providing you expert opinion on what you need to know to start your own meta-analysis.

► With the thousands of meta-analyses conducted in all areas of psychology over the past few decades, there has been an ever-increasing number of articles, books, and software programs devoted to how to conduct meta-analyses. Below, experts on meta-analysis provide their suggesstions on which which of the many sources of information are the most useful and why -- so that the user has an easy-to-use starting place for learning everything about meta-analyses.



Contents



Where should I start?

If you want to learn what is a meta-analysis...

  1. For the basics, see below where we lay out:

  2. For more in-depth discussion and explanations, we recommend...
    • start first with (Johnson & Eagly, 2000) which is a chapter from the "Handbook of Research Methods in Social and Personality Psychology" that provides an excellent detailed explanation of each stage in the meta-analysis process including the advantages of meta-analysis compared to traditional literature reviews, statistical forumlas for analyzing effect sizes from various indexes and study designs, and discussion of how to visually depict your data.
    • then for even more in-depth descriptions see (Cooper & Hedges, 1994) (Handbook of Research Synthesis) which provides a separate chapter on every step involved in designing, analyzing, and writing-up a meta-analysis.

If you want to learn how to start conducting a meta...

  1. For the basics, see below were we lay out:

  2. For more in-depth discussion and explanations, we recommend...
    • start first with (Johnson, Mullen, & Salas, 1995) which provides a statistical comparision of the three major meta-analytic approaches using actual datasets, as well as the staistical forumulas for each approach and the methodological differences between each approach.
    • based upon which meta-analytic approach you choose to use, see (Lipsey & Wilson, 2001) for the Hedges/Olkin approach, see (Rosenthal, 1991) for the information on the Rosenthal/Rubin approach, or see (Hunter, Schmidt, & Jackson, 1982) for the Hunter/Schmidt/Jackson approach.



What is a meta-analysis?

Definition

A meta-analysis statistically combines the results of several studies that address a shared research hypotheses.

Just as individual studies summarize data collected from many individual participants in order to answer a specific research question (e.g., each participant is a separate data point), a meta-analysis summarizes data from individual studies that concern a specific research question (e.g., each data point is each individual study).

The results of each individual study are converted to a standardized effect size. A meta-analysis combines...


Three Basic Questions

A meta-analysis answers three general questions:
  1. Central tendency – The central purpose of a meta-analysis is to test the relationship between two variables such that X causes Y. Central tendency refers to identifying whether X affects Y via statistically summarizing signficance levels, effect sizes, and/or confidence intervals. You are trying to answer whether X affects Y, is the effect significant, and how strong is that effect?
  2. Variability – There is always going to be some degree of variation between the outcomes of the individual studies that compose the meta-analysis. The question is whether the degree of variablity is signficantly different than what we would expect by chance alone. If so, then its called heterogeneity.
  3. Prediction – If there is heterogeneity (variability), then we look for moderating variables that explain the variability. In other words, does the effect of X on Y differ with moderator variables?

Five Basic Steps

There are generally five separate steps in conducting a meta-analysis:
  1. Define your Hypothesis – First you must have a well-defined statement of the relationship between the variables under investigation so that you can carefully define the inclusion and exclusion criteria when locating potential studies. For more information see Chapter 2 of (Lipsey & Wilson, 2001) (Practical Meta-Anlaysis) for a thorough examination of this step.
  2. Locate the Studies – A meta-analysis is only informative if it adequately summarizes the existing literature, so a thorough literature search is critical to retrieve every relevant study, such as database searches, ancestry approach, descendancy approach, hand searching, and the invisible college (e.g., network of researchers who know about unpublished studies, conference proceedings, etc). For more information see (Johnson & Eagly, 2000) (Handbook of Research Methods in Social and Personality Psychology) which details five general ways to retrieve relevant articles.
  3. Input data – Gather empiricial findings from primary studies (e.g., p-value, effect size, etc) and input into statistical database. Not every study provides sufficient statistical information to calculate the effect size statistic. For more information see below about choosing your statistical software.
  4. Cacluate Effect Sizes – Calculate the overall effect by converting all statistics to a common metric, making adjustments as necessary to correct for issues like sample-size or bias, and then calculating central tendency (e.g., mean effect size and confidence intervals around that effect size) and variablity (e.g., homogeneity/heterogeneity analysis). For more information see below about choosing which effect size to calculate and see any meta-analysis book for all the statistical formulas.
  5. Analyze Variables – If heterogeneity (variability) exists, you may want to analyze moderating variables by coding each variable in the database and analyzing either mean differences (for categorical variables) or bivariate prediction (for continuous variables) to see if the variable accounts for the variability in the effect size.

Two (x Two) types of Variables

In addition to variables being categorical (e.g., discrete categories) or continuous (e.g., assigned Likert scale value), ....

There are two types of study variables: (a) objective variables -- such as type of IV or DV, ..., (b) subjective variables -- inferential judgements made by two or more judges....

such as publication type (e.g., articles, books, dissertation, technical reports, unpublished, etc) design features (e.g., experimental, correlational, random assignment control group

How do I conduct a meta-analysis?

First, choose which statistical approach suits your needs

There are generally three different statistical approaches to conduct a meta-analysis so first you need to choose which approach best fits your needs. For an excellent detailed comparison of these three approaches, see (Johnson, Mullen, & Salas, 1995) (Comparison of Three Major Meta-Analytic Approaches. Journal of Applied Psychology, 80, 94-106). Some basic information from that article is posted below to get you started:
  1. Hedges & Olkin Approach – see (Hedges, 1981); (Hedges, 1982); (Hedges & Olkin, 1985)
  2. Rosenthal & Rubin Approach – see (Rosenthal, 1991); (Rosenthal & Rubin, 1978); (Rosenthal & Rubin, 1988)
  3. Hunter, Schmidt, & Jackson - see (Hunter, Schmidt, & Jackson, 1982); (Hunter & Schmidt, 1990)
Table1JMS2.gif

Second, choose which effect size to calculate?

asdfadsfsdf

Third, choose your statistical software

adsfsdafds

see "practical meta analysis chapter 5 (page 91) which discusses pros and cons of different ways to... including...

DSTAT calculates all of this for you Can also use SPSS and macros from “Practical Meta-Analysis” Calculate Categorical variables – DSTAT using weighted ANOVA Calculate Continuous variables – SPSS using weighted Regression


If you want more detailed information about...

...the Hedges & Olkin approach...

...the Rosenthal & Rubin approach...

...the Hunter, Schmidt, & Jackson approach...













► Back to Research Tools mainpage

Personal tools
Namespaces
Variants
Actions
Navigation
Interaction
Toolbox