Other questions about meta-analyses?
From PsychWiki - A Collaborative Psychology Wiki
Step 2 - Locating Studies
How long should searching for articles take?
- It is to your advantage to conduct the searching for articles (Step 2) in a compact period of time because you will invariably come across the same abstract over and over again when using different search terms, so if you conduct the search in a compact period of time it will be easier to remember which abstracts you have already read (and decide to pass or keep)
Why should I email listservs?
- Emailing the relevant listserv for articles that are unpublished is another good way to update your search and find relevant works. Plus, you are in a sense “marking your territory” for that topic so it lets other researchers avoid wasting their time conducting the same meta analysis. Conversely, emailing the listserv will tell you if someone else is already doing the meta analysis.
Step 3 - Finding ES
What to do about the issue of dependence of effect sizes?
- Should you compute more than one ES per study? Best solution is to compute both ways. In some cases you might want to have data from same Ss be in two or more effect sizes. For example, if want to look at different types of DVs in relation to overall effect sizes, then if same Ss in same study filled out two or more DVs, then AFTER doing normal independent overall ES (where you only take 1 ES per study), do a second stage test of non-independence and coding more than one ES per Ss. (See Johnson and Eagely, 2000, page 519
If only taking 1 ES per study, how to choose which one?
- Either pick the one that is most common across the studies. Or, pick the average of the ES within the study.
What if the primary study reports the effect size directly?
- Should you use it, or should you try to compute the ES yourself based upon the statistical information from the article? First, try computing the ES yourself, and see how it compares to the ES reported in the study. Are they similar? However, keep in mind that some articles do not report enough statistical information to calculate the ES yourself, in which case you have to use the ES reported by the authors. Also, sometimes when computing the ES yourself from the statistical information in the article you encounter situations where you have imprecise statistical information, such as the authors reporting p < .05, in which you case must treat that as p = .05. In such a case, the ES you compute will be imprecise, and thus you may want to use the ES reported by the authors.
Step 4 - Analyzing Data
Should I use fixed or random?
- Start by looking at the output for “fixed. If “random” is significant, then means that there is variation not explained by moderators.
How to interpret the Effect Sizes?
- See Johnson and Eagley (2000) Quantitative synthesis.. page 520. see also Lipsey and Wilson (2001) Practical meta-analysis, page 146-
Step 5 - Moderators
How many moderators can I anlayze?
- Meta is exploratory, so you can code for as many variables as you wish. Here is citation to support that statement (Guzzo, Jackson, and Katzell, 1987). Guzzo, R.A., Jackson, S.E., & Katzell, R. A. (1987). Meta-analysis analysis. In L. L. Cummings and B. M. Staw (eds.), Research in Organizational Behavior, Vol. 9, pp. 407-442. Greenwich, CT: JAI Press.
Should the moderators be categorical or continuous?
- Ideally, continuous. Why? Continuous scale ranges contain more data, so more power. Plus, you can convert a continuous variable into a categorical variable, but you can't do the reverse. Sometimes the variable itself dictates whether it is categorical or continuous. For example, gender as a moderating variable is categorical by design.
Why should the coding sheet be in excel?
- When creating the “coding sheet” that you give to coders and upon which they enter their codes, it is easiest for you if the coding sheet is an e-copy, such as excel, because the data is therefore already entered.
Why would there be inconsistency in coders ratings?
- Why would there be inconsistency? It may be a coding mistake. It may be because the coders are misunderstanding how to code the variable. You may want to have a third-party tie-breaker decide the issue. Try to not let the coder with the most dominant personality dictate how the issue is resolved. Some inconsistency is expected, which is why there are multiple coders.
How to increase quality of coders/codings?
- Since coding can be boring, and there is a temptation to rush through the coding too quickly, impress upon the coder the seriousness of their impact on the study (because garbage-in garbage-out). You may also try impressing upon them the consequences of poor performance and how you are double checking their work (somehow?)