Introduction to Meta-Analysis

What is a Meta-Analysis?
Why Do We Need Meta-Analyses?

Ahmet H. Kirca, Ph.D.

Associate Professor
Michigan State University

What is a Meta-Analysis?

A statistical analysis that combines or integrates the

results of several independent clinical trials considered
by the analyst to be combinable ASA, 1988
now widely accepted as a method of summarizing the
results of empirical studies within the behavioral, social
and health sciences Lipsey and Wilson, 2000
Meta-analysis is a quantitative technique that help a
field to take stock of knowledge in a particular domain
by systematically combining and synthesizing findings
across studies.

1
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?

T.S. Eliot, The Rock

Why Meta-Analysis?
Science is a cooperative and cumulative enterprise.
Thousands of new scientific studies completed
every year but we have limited time to find,
evaluate, apply these new studies in our research
and practice
Evidence-based decision making is taking hold in
medicine, education, marketing, management,
political science and other fields
Qualitative reviews and vote counting methods are
inadequate in various ways

2
 King of the hill in health sciences

Useful for Teaching and Practice

Webinar: D. Hanssens (UCLA) presents highlights from the book

http://www.msi.org/video/empirical-generalizations-about-marketing-impact/

3
 The Great Debate
1952: Hans J. Eysenck concluded that there were no
favorable effects of psychotherapy, starting a raging
debate
20 years of evaluation research and hundreds of studies
failed to resolve the debate
1978: To proved Eysenck wrong, Gene V. Glass
statistically aggregate the findings of 375 psychotherapy
outcome studies
Glass (and colleague Smith) concluded that
psychotherapy did indeed work
Glass called his method meta-analysis

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Emergence of Meta-Analysis

Ideas behind meta-analysis predate Glass work by
several decades
Karl Pearson (1904)

averaged correlations for studies of the effectiveness of

inoculation for typhoid fever
R. A. Fisher (1944)
When a number of quite independent tests of significance
have been made, it sometimes happens that although few or
none can be claimed individually as significant, yet the
aggregate gives an impression that the probabilities are on the
whole lower than would often have been obtained by chance
(p. 99).
Source of the idea of cumulating probability values

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

8

4
 Meta-Analysis: International Business, Marketing, and
Management Disciplines between 1980-2009
Top 25 Top 25
International Mainstream Top 25
Journal Business Number of Marketing Number of Management Number of
Ranking Journals1 Meta-Analyses Journals2 Meta-Analyses Journals3 Meta-Analyses
1 JIBS 5 JM 7 JAP 177
2 MIR 1 JMR 19 MNS 4
3 JWB 0 JCR 8 AMJ 24
4 IMR 3 HBR 0 AMR 4
5 ISMO 0 MNS 4 ASQ 2
6 JIM 0 ACR 0 SMJ 6
7 IBR 0 MKS 4 OBHDP 16
8 AICM 0 JAMS 8 JVB 19
9 AIM 0 JR 5 PerPsych 66
10 JIFMA 0 IMM 0 HBR 0
11 JGM 1 JAR 5 HR 11
12 IJRM 6 JBR 11 I&LRR 3
13 AIA 0 JIBS 5 JOM 19
14 IJA 2 SMR 0 JHR 1
15 IM 0 JA 2 DS 1

Number of Meta-Analyses by Year

35

30
N u m b e r o f m e ta -a n a ly s e s

25

IB Journals
20
Management Journals

15 Marketing Journals

10

0
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
198
198
198
198
198
198
198
199
199
199
199
199
199
199
199
199
199
200
200
200
200
200
200
200
200
200
200

Years

5
 The Logic of Meta-Analysis
Traditional methods of review focus on statistical
significance testing
Significance testing is not well suited to this task
Highly dependent on sample size
Null finding does not carry the same weight as a significant
finding
significant effect is a strong conclusion
nonsignificant effect is a weak conclusion
Meta-analysis focuses on the direction and magnitude
of the effects across studies, not statistical significance
Isnt this what we are interested in anyway?
Direction and magnitude are represented by the effect size

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

11

Types of Systematic Reviews: Qualitative Vs.

Quantitative
Qualitative systematic Quantitative systematic
review (best evidence review (meta-analysis)
synthesis) The results of two or
The results of primary more primary studies are
studies are summarized combined
Not statistically Results are statistically
combined examined
Described narratively Use methods to limit
Still use other methods bias
to limit bias

6
 Strengths of Meta-Analysis

Imposes a discipline on the process of summing up

research findings
Represents findings in a more differentiated and
sophisticated manner than conventional reviews
Capable of finding relationships across studies that are
obscured in other approaches
Protects against over-interpreting differences across
studies
Can handle a large numbers of studies (this would
overwhelm traditional approaches to review)

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

13

Weaknesses of Meta-Analysis
Requires a good deal of effort
Mechanical aspects dont lend themselves to capturing
more qualitative distinctions between studies
Apples and oranges criticism
Most meta-analyses are correlational in nature
Selection bias posses a continual threat
Negative and null finding studies that you were unable to find
Outcomes for which there were negative or null findings that were
not reported
Analysis of between study differences is fundamentally
correlational

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

14

7
 Discussion

15

Assignment 1

16