Meta Analysis 2019 2020

M. Sc.
Research in Behavior &

Cognition
Workshop on data analysis
1. Integrating results in meta-analysis
1. Some history
2. Systematic review vs. meta-analysis
3. Definition and objectives of meta-analysis
2. Steps in meta-analysis
1. Formulating the problem
2. Searching the literature
3. Coding
4. Computing effect sizes
5. Combining effect sizes: weighted effect size, statistical models and homogeneity test
6. Moderators analyses: MetaAnova and Metaregression
7. Exploring biases
8. Interpreting the evidence and presenting the results
3. Advanced topics in meta-analysis
4. Metafor: R package for meta-analysis
1. Some history
2. Systematic review vs. meta-analysis
3. Definition and objectives of meta-analysis

➢ Science is a cumulative process
➢ Many studies address the same basic question
➢ Researchers try to summarize the literature on a particular

topic
➢ There are several strategies, e.g., narrative review, scoping

review, systematic review, meta-analysis
➢ 1952 --- Hans J. Eysenck concluded that there were no favourable

effects of psychotherapy, thereby starting a raging debate
➢ 20 years of research and hundreds of studies failed to resolve the
debate
➢ 1977 --- Gene V. Glass statistically aggregated the findings of 375
psychotherapy outcome studies
➢ Glass (and colleague Smith) concluded that psychotherapy did
indeed work
➢ Glass called his method “meta-analysis” Gene V. Glass
Hans J. Eysenck
1. Integrating results in psychology
➢ After him… Olkin, Cohen, Rosenthal, Rubin, Hunter, Schmidt,

Raju, Hedges, Cooper, etc.
From Guilera et al. (2012)

➢ Systematic review is a revision of the literature that requires

protocols, transparency, and rigor like individual studies to
integrate findings on a given topic:
➢ Objectivity
➢ Replicability
➢ Systematization
➢ Meta-analysis is a systematic literature review supported by

statistical methods whose goal is to aggregate and contrast
findings from several related studies (Glass, 1976)
Meta-analysis = Systematic review + quantitative estimation of effects

The streptokinase example

➢ From 1959 to 1988 there were 33 RCTs to assess the capacity of
streptokinase to prevent death following a heart attack
➢ Streptokinase was supposed to increase the likelihood of survival
➢ All the trials followed similar protocols (random assignment to treatment or

placebo)
➢ The outcome, whether or not the patient died, was the same in all the studies
➢ The trials varied substantially in size: Md ≈ 100; 2 large scale trials (12,000
and 17,000 patients)
➢ 6 studies were statistically significant in favour to streptokinase while the

other 27 were not
6 were statistically
significant (look at CI; they
do not include 1.0), 27 were
not (they had small sample
sizes and low statistical
power)
Conclusion of a narrative
or systematic review:
evidence against effect or
inconsistent results
Conclusion of MA: Effect
reasonably consistent
(most between 0.50 and
0.90) – appropriate
composite a summary
effect. The summary effect
is a RR of 0.79 (95% CI
0.72-0.87) – 21%
decrease in risk of death.
The p-value is 0.00000008
Those treated were more likely to survive
➢ Meta-analysis is a systematic literature review supported by

statistical methods whose goal is to aggregate and contrast
findings from several related studies (Glass, 1976)
➢ Objective: Summarize the empirical research on a topic
➢ Offer a guide to practitioners regarding intervention

PRACTICE
(comparative) effectiveness
➢ Provide an updated view on a given topic
➢ Enhance establishing the generality of findings + identifying
SCIENCE
relevant moderator variables

➢ Replicated evidence as an indicator of significance (incl. p)
1. Formulating the problem
2. Searching the literature
3. Coding
4. Computing effect sizes
5. Combining effect sizes: weighted effect size, statistical
models and homogeneity test
6. Moderators analyses: MetaAnova and Metaregression
7. Exploring biases
8. Interpreting the evidence and presenting the results
Step 1. Formulating the problem
➢ Careful statement of the topic to be studied or the

question to be answered
➢ This statement will guide the selection of studies, the

coding of information from those studies, and the
analysis of the resulting data
➢ The statement should be complete, but initially need

not be highly detailed
Step 1. Formulating the problem
➢ Identify the form(s) of research findings relevant to the topic of

the meta-analysis and ensure that they can be represented with a
common ES statistic
What proportion of women suffer from
Central migraines?
tendency What is the mean number of days in
description hospital of first-episode schizophrenics?
How much did children’s reading
scores improve at the end of the
school year?
Mean number of aggressive Pre-post Effect Group
behaviour episodes before and after a contrasts size? contrasts
prevention program
How effective was a treatment in the

experimental group compared to the
Association control group?
What is the relationship between alcohol between Are there differences between men
use and domestic violence? variables and women in prosocial behaviour?
What is the test-retest reliability of STAI?
Step 2. Searching the literature
➢ Significant findings are more likely to be published than

non-significant findings – Publication bias
➢ Searching far and wide – Grey literature – Quality of

studies? – The “we only included published studies
because they have been peer-reviewed” argument
➢ Go broad!
▪ Computerized bibliographic databases – which keywords?

▪ Review articles
▪ References in studies
▪ “Google” internet search engine
▪ Hand searching in relevant journals
▪ Conference programmes and proceedings
▪ Authors and experts
▪ Dissertations
▪ Government reports, bibliographies, clearing houses
➢ Electronic searches:
➢ Type I error: broad searches normally retrieve a large number of studies
that no human could summarize
➢ Type II error: narrow searches routinely miss relevant studies
➢ Identifying inclusion/exclusion criteria:

➢ Better to be done by 2 experts
➢ Search for duplicates
➢ Difficult to retrieve all studies that meet your eligibility

criteria
➢ Excel is a good option, but there are specific tools

➢ Key point: Clearly explain your search strategy,

inclusion/exclusion criteria, and show a flow
chart or flow diagram when you carry out (and
publish) your MA
Image extracted from the APA Style Blog:

http://blog.apastyle.org/apastyle/2013/10/ho
w-do-i-cite-a-search-in-apa-style.html
➢ Think about a topic you are interested in that would be

susceptible to meta-analysis
➢ Try to select your keywords, define your search strategy, choose

the databases, and think of other resources for your search
➢ Search in one of the databases
➢ How many reports did you find?

Step 3. Coding
➢ Characteristics of studies
➢ How the results are reported
➢ Characteristics of the setting and subjects
➢ Data on treatments used (if relevant)
➢ Study outcomes Potential

MODERATORS
Step 3. Coding
➢ Development of coding protocol (manual + form)

 Essential feature of meta-analysis
 Transparent and reproducible
 description of studies
 extraction of findings
 Dynamic process – the first draft should never be the last
 PRISMA-Protocol
➢ Training coders
➢ Assessing the accuracy of extracted information

Step 3. Coding
Information Details
Study level information Report information – e.g., ID, type of publication, year of
publication, funding, etc.
It needs to be coded only once Substantive issues– e.g., sample descriptors (age, sex,
for a given study and does not diagnostic, etc.), type of treatment, dose, theoretical
vary for different outcomes, orientation, etc.
follow-ups, sample breakdowns,
etc. Methods and procedures – e.g., sampling method, quality
of measures, follow-up measures, etc.
Effect size level information Dependent measure descriptors – e.g., type of ES,
outcome construct, etc.
Aspects that are specific to a ES (numerical) data – e.g., means and SD, ES based on,
particular quantitative relationship favoured group, calculated ES, page number where ES
or study finding was extracted, etc.
Step 3. Coding
Manual Form
From Lipsey and Wilson (2001)

Step 3. Coding
Manual
Form

Step 3. Coding
 First session: coding manual and form (paper,

spreadsheet, MA software, etc.) explanation and
practice coding
 Regular meetings - develop normative understanding
 Write comments on the coding manual
 “Specialist” coders
Step 3. Coding
 Be trained (manual + form)

 Complete form for all studies retrieved as potentially
eligible
 Write comments when necessary
 Modify criteria after examining sample of studies
 When possible, double-code study eligibility and study
data (at least part of it)
 Maintain database of results for each study screened
(reasons for exclusion, etc.)
Step 3. Coding
Paper MA software
Example of
FileMaker
Pro from
Lipsey and
Wilson (2001)
Spreadsheet
Example of
Excel from
Example of paper coding Borenstein et
from Cooper (2010) al. (2009)
Step 3. Coding
 Several constructs – e.g., treatment effects on social skills,

self-esteem and employment status
 Several measures – e.g., BDI and HAM-D for assessing
depression
 Several samples – e.g., relationship between anxiety test
and maths grades in boys and girls
 Several times of measurement – e.g., treatment effects
assessed at 2 weeks, 3 months and 6 months
Single ES
Hierarchy of ES
per study
Step 3. Coding
➢ Example SGA vs Haloperidol

Multiple Several scales:
Several comparisons: Several outcomes: Several time points:
studies RAVLT, List of words short-term, medium-
within an H vs Clozapine, H vs WM, attention, verbal (WMS), COWAT for term, long-term
article Risperidone, etc. learning, etc.
verbal learning
From Guilera et al. (2009)

Step 3. Coding
➢ Strategies
 Average
 Select one via a substantive criterion
 Sample one at random

Step 3. Coding
➢ Consistency (≥ 20% documents)
 Intra-rater reliability – consistency of a single coder

from occasion to occasion
 Inter-rater reliability (double coding) – consistency

between different coders
 Percentage of agreement, kappa, r, ICC, etc.
 Disagreement – consensus or third coder

Step 3. Coding
Step 3. Coding
 Think about a topic of interest to you
 Draw up a preliminary coding guide - choose 5 variables from

sample characteristics, intervention, etc.
 Find 3 research reports relevant to the topic
 Apply the coding form to these studies
 What did you learn about your problem definition and

the research methods used in attempts to study it?
From Cooper (2009)
Step 3. Coding
 In a published MA…
 Look at the search strategy. Do you think it is replicable?
 Do the authors specify inclusion and exclusion criteria?
 Do the authors show a flow chart?
 Do the authors use any strategy to keep a single ES per study?
 Is there any measure of agreement between coders presented?

Which one?
Step 4. Computing effect sizes
Definition Quantitative measure of the

strength of a phenomenon
Aim Summarize the results of a study
and enable further summaries
Examples Pearson’s r, Cohen’s d, OR, eta2,

etc.

➢ Treatment effects – Medicine – OR, RR, or RD typically used in

medical interventions
➢ Effect size – Social sciences – SMD or correlations typically used

in social science MA
➢ Effect size includes treatment effects but it can include other

effects (e.g., difference between boys and girls, relation between
intelligence and academic marks, etc.)
➢ Single group summary – mean, risk, rate in a single population –

not effect size since effect implies a relationship
➢ The effect size (ES) makes meta-analysis possible
➢ The ES encodes the selected research findings on a

numeric scale
➢ There are many different types of ES measures, each

suited to different research situations
➢ Each type of ES may also have multiple methods of

computation
➢ The type of ES must be the same across studies
➢ The ES chosen must be appropriate for the nature of

the relationship described in the selected research
findings
➢ It must also be appropriate for the statistical forms in

which those findings are reported
➢ I will focus on ES based on continuous, binary and

correlational data
➢ ES from different studies should be comparable with each other

– they measure the same thing
➢ Estimates of the ES should be computable from the information

that is likely to be reported in published research
➢ The ES should have good technical properties – the sampling

distribution should be known so that variances and CI can be
computed
➢ The ES should be substantively interpretable – potential readers

should find the ES meaningful
In practice... The kind of data used in the primary studies will usually
lead to a pool of 2 or 3 ES that meet the criteria:
Primary study data Primary study data are Primary study

are based on based on binary reports a correlation
means and SD in outcome such as between two
two groups events or non-events variables
in two groups
RMD, SMD, etc. Correlation

OR, RR, RD, etc.
➢ Independence of ES – each study should contribute the same to the

analysis – statistical models assume independence of ES
➢ Inverse variance weight

◼ Problem: some ES are more accurate than others
◼ An ES based on 100 subjects is assumed to be a more “precise” estimate
of the population ES than an ES based on 10 subjects
◼ Therefore, larger studies should carry more “weight” in our analyses than
smaller studies
◼ The standard error (SE) is a direct index of ES precision
◼ Hedges and Olkin showed that the optimal weights for meta-analysis are:
1 1 It provides statistical basis for SEM, 95% CI,

w= = homogeneity testing, Z-value to test H0
V SE 2
Unstandardized (raw) mean difference (RMD) - D
The outcome is reported using a meaningful scale and Blood pressure

ALL studies use the same scale PANSS
Two independent groups Two dependent groups

D = X diff
D = X1 − X 2 Difference
2
S diff scores are
n +n 2 VD =
VD = 1 2 S pooled assuming σ1 = σ2 Where n is the
reported
n1n2 n number of pairs
S pooled =
(n1 − 1)S12 + (n2 − 1)S 22 D = X1 − X 2 Correlation
Pre- and
post-scores
n1 + n2 − 2 2
S diff
between pre- and
VD =
post-scores are reported
2 2
S S n
VD = 1
+ 2 assuming σ1 ≠ σ2
n1 n2 S diff = S12 + S 22 − 2  r  S1  S 2 assuming σ1 ≠ σ2
S diff = 2  S pooled
2
(1 − r ) assuming σ1 = σ2
Standardized mean difference (SMD) – d or g
The outcome is reported using a non-meaningful BDI and HAM-D

List of words
scale or studies use different scales of measurement (WAIS) and RAVLT
Two independent groups Two dependent groups Cohen’s d is biased

when sample size is
small
X1 − X 2 Ydiff Y1 − Y2
d= d= =
S within S within S within
S within =
(n1 − 1)S12 + (n2 − 1)S 22 S diff
S within =
n1 + n2 − 2 2(1 − r ) Hedges’ g
n1 + n2 d2 1 d2 
Vd =  + 2(1 − r )
Vd = + 3
J = 1−
n1n2 2(n1 + n2 )  n 2n  4df − 1
n1 + n2 − 2 → Ind .Groups g = J d
n − 1 → MatchedGroups Vg = J 2  Vd
The direction of the

effect is arbitrary, but
should be applied
consistently
Time to complete TMT-B

+ ES
Pairs recalled
+ ES
From Purdon et al. (2001)

Cohen’s criteria (1988)
SMD = 0.20 – Small
SMD = 0.50 – Medium
SMD = 0.80 - Large
From Littell et al. (2008)

Choose an outcome and compute:

• D and its variance
• d and its variance
• g and its variance
Events Non-Events N
Treated A B n1
Control C D n2
Dead Alive N
Treated 5 95 100
Control 10 90 100
Risk Ratio (RR)
This is the ratio of two risks. It is intuitive and has a

clear meaning. Calculations use a log scale.
A n1
RR =
C n2
LogRR = ln(RR ) RR = exp(LogRR)
1 1 1 1
VLogRR = + + +
A n1 C n2
Odds Ratio (OR)
This is the ratio of two odds. It is less intuitive, but

has very useful statistical properties. Calculations use
a log scale.
A B AD
OR = =
C D BC
LogOR = ln(OR ) OR = exp(LogOR)
1 1 1 1
VLogOR = + + +
A B C D
Risk Difference (RD)
This is the difference between two risks. It is intuitive.

Calculations use raw units.
 A  C 
RD =   −  
 n1   n2  NO log transformation
AB CD
VRD = 3 + 3
n1 n2
Dead Alive N
Treated 5 95 100
Control 10 90 100
The risk of dying in the 5 95 5  90

treated group is half the OR = = = 0.47
10 90 95 10
risk of dying in the control
RR =
5 100
= 0.50 group LogOR = ln(0.47) = −0.75
10 100
1 1 1 1
LogRR = ln(0.50) = −0.69 VLogOR = + + + = 0.32
5 95 10 90
1 1 1 1 The odds of death in the treated
VLogRR = + + + = 0.28
5 100 10 100 group is half the odds of death
in the control group
 5   10 
RD =  −  = −0.05 The difference between the
 100   100  risk of dying in the treated
5  95 10  90 group and the risk of dying in
VRD = 3
+ 3
= 0.001 the control group is -0.05
100 100
Choose an outcome and calculate:

•RR and its variance in log units
•OR and its variance in log units
•RD and its variance in log units
From Littell et al. (2008)

Correlation coefficient (r)
Studies report a correlation between two continuous Relation between

variables. It is intuitive. It is a standardized measure. quality of life and
cognitive functions
1+ r 
z = 0.5  ln 
1− r  e2 z −1
r = 2z
1 e +1
Vz =
n−3
Vr =
(1 − r )
2 2
n −1
Choose an outcome and calculate:

•Fisher’s z and its variance
➢ The Metafor package provides the escalc ( ) function

➢ It calculates ES or outcome measures (and the corresponding
variances) that are commonly used in MA
are used to supply the information needed to calculate

character string specifying which
the various measures (depending on the outcome
outcome measure should be
measure specified under measure, different arguments
calculated
need to be supplied)
to specify a data arguments

needed when dealing sampling variance estimate that
frame containing the should be calculated
variables given to the with 2 × 2 data table
previous arguments that may contain
cells with zeros When setting append = TRUE, the data frame specified
via the data argument is returned together with the
effect size estimates and corresponding sampling
variances.
➢ The most common options for the measure argument are:
➢ “RR“: log relative risk

➢ “OR“: log odds ratio
➢ “RD“: risk difference
➢ “MD“: raw mean difference
➢ “SMD“: standardized mean difference (Hedges' g)
➢ “COR“: raw correlation coefficient
➢ “ZCOR“: Fisher’s r to z transformation for correlation
coefficient
➢ “PR“: raw proportion
➢ “PLO“: logit transformed proportion
➢ In binary data, the arguments...
Cell entries with a 0 can be problematic for the RR and OR. Adding
a constant to the cells of the 2 × 2 tables is a common solution.
Usually 0.5 is added to each cell of the 2 × 2 table only in those
tables with at least one cell equal to 0.
➢ In continuous data, the arguments...
➢ m1i and m2i: the means of group 1 and 2, respectively
➢ sd1i and sd2i: the standard deviations of group 1 and 2,

respectively
➢ n1i and n2i: the sample sizes of group 1 and 2, respectively

➢ In correlational data the arguments...
➢ ri: the raw correlation coefficient

➢ ni: sample size
➢ In summary effects data (dichotomous) the arguments...
➢ xi: the number of subjects experiencing the event

➢ ni: total number of subjects (or mi: the number of subjects that
do not experience the event)
➢ Example
➢ dat.bcg – 13 studies of the effectiveness of the BCG vaccine
against tuberculosis (Colditz et al., 1994)
number of treated
(vaccinated) subjects that were
tuberculosis positive
Note. log RR < 0 indicates a number of control (non-

lower infection risk for the vaccinated) subjects that were
vaccinated group tuberculosis negative
Let us move to R…
➢ Example
➢ Example_means – 15 studies of the differences between men
and women in maths anxiety
trial author wm wsd wn mm msd mn
1 Carroll 94 22 60 92 20 60
2 Grant 98 21 65 92 22 65
3 Peck 98 28 40 88 26 40
4 Donat 94 19 200 82 17 200
5 Stewart 98 21 50 88 22 45
6 Young 96 21 85 92 22 85
Let us move to R…
Situation 1. Some papers do not report the basic information

needed to compute the ES
Example...
Many formulas to compute ES from other information, such as t

values, F values, etc.
Situation 1. Some articles do not report the basic information

needed to compute the ES
In Lipsey & Wilson (2001) there are almost 40 formulas for

calculating ES from a range of statistical data
n1 + n2
Example SMD = t
n1n2
Practical meta-analysis ES calculator (D. Wilson)

http://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-Home.php
HLS-Meta (Huedo-Medina et al., 2013)
https://taniabhuedomedina.alliedhealth.uconn.edu/research-tools/
Situation 2.
Some articles report an ES measure and others use another ES
measure
Think about the appropriateness of combining such

studies
It only makes sense to compute a summary effect

from studies that are comparable
There are many formulas to convert between ES measures, such as

from LogOR to d, from r to d, etc.
Rules of thumb (Cohen, 1988)

➢ Think about a topic you are interested in
➢ Look for 2 papers susceptible to meta-analysis
➢ Choose an ES measure you think is meaningful
➢ Extract the data for computing an ES measure
➢ Compute the corresponding ES and its variance

 In a published MA…
 What ES measure is used?
 Do the authors use any ‘transformation formula’?
 What criteria do the authors use to interpret ES?
 Which software do the authors use?

Step 5. Combining effect sizes
Individual studies
Effect size (ES)
Precision
Study weights
p-values
Summary effect
Effect size (ES)
Precision
p-value
Heterogeneity of ES
From Borenstein et al. (2009)

What is the summary effect?

Is this effect consistent across studies?
Fixed-effect model (FEM) Random-effects model (REM)

There is one true ES which The true effect could vary from
underlies all the studies study to study (e.g., older
All differences in observed ES participants, more educated, longer
(between-studies variation) are due to interventions, etc.)
sampling error True ES are normally distributed
Weights: within-study variance Variability in ES is due to sampling
error + variability in the true effects
Weights: within and between-study
variance (τ2)
Fixed-effect model Random-effects model
Population effect sizes Population effect sizes
From Huedo-Medina workshop (2013)

➢ Fixed-effect: less weight

to smaller studies as they
are less informative about
the common effect size.
➢ Random-effects: more
balanced weights (due to
τ2) not to misrepresent any
of the true effect sizes.
To estimate the population effect weighted mean

weight assigned
Summary effect and its variance to each study is
the inverse of its
k variance
W Y i i
weighted mean M= i =1
k
W
i =1
i
1
Wi =
1 VYi
VM = k
weighted variance
W
i =1
i within-study variance
SEM, 95% CI, Z-value to test H0: θ = 0

(for a difference) or θ = 1 (for a ratio)
To estimate the population effect weighted mean

Tau-squared (Method of weight assigned
Summary effect
moments, DerSimonian- to each study is
and its variance
Laird)
the inverse of
Q − df
T2 = VY*i = VYi + T 2 variance
C
k
W Y
2
 k
 *
  WiYi  i i
M =* i =1
W Y 2 −  i =1 
k
Q= k
 W i*
1
W i* =
i i k
i =1
W i =1
i i =1 VY*i
1
df = k − 1 VM * = k
within-study variance +
 i
W 2
W *
between-study variance (τ2)
C =  Wi −
i
i =1
W i
SEM, 95% CI, Z-value to test H0: θ = 0
(for a difference) or θ = 1 (for a ratio)
➢ FEM gives much more weight to studies with higher sample size
than REM – the weights under the REM are more balanced
➢ CI are narrower with FEM – effects that are significant under a
FEM may no longer be significant under a REM – REM more
conservative
➢ REM become FEM when distributions are homogeneous (tau-
squared = 0)
➢ Assumptions of FEM rarely plausible
➢ Historically, most MA in Psychological Bulletin have used FEM
➢ General advise: use REM a priori
➢ Area of active debate among statisticians
The Metafor package can fit FEM and REM

ES and its variance are used to supply the information needed to calculate
or SE the various measures
Method used to To deal with 0 in 2 × 2 sampling

estimate tau-squared When we tables variance
supply yi and To specify data estimate that
vi (or sei), we file should be
must set this calculated
variable
“GEN”, the
default
For a FEM:
method = “FE”
➢ Example
➢ dat.bcg – 13 studies of the effectiveness of the BCG vaccine
against tuberculosis (Colditz et al., 1994)
number of treated
(vaccinated) subjects that were
tuberculosis positive
Note. log RR < 0 indicate a lower

number of control (non-
infection risk for the vaccinated
group
vaccinated) subjects that were
tuberculosis negative
Let us move to R…
The risk of tuberculosis
infection in vaccinated
individuals is on average
Fitting a REM (method = REML by default) half as large as the infection
risk without the vaccination
Transformed to RR = exp
(-0.7145) = 0.49 (95% CI:
0.34 – 0.70)
With the data: Example_correlations
1. Check the data

2. Combine the data under the REM
3. What is the mean ES? Is it statistically significant?
4. What is the value of τ2?
5. Check that the CIs are narrower under the FEM
Send the answers via Campus Virtual

(Document with all the results and explanations)
➢ Homogeneity analysis tests whether the assumption that all of the

ES are estimating the same population mean is a reasonable
assumption
➢ If homogeneity is rejected, the distribution of ES is assumed to

be heterogeneous
▪ Single mean ES not a good describer of the distribution
▪ There are real between-study differences; that is, studies estimate different
population mean effect sizes
▪ Random effects model addresses this issue
▪ You can also explore this excess of variability with moderator analysis
▪ Q is simply a weighted sum-of-squares:

k
Q = Wi (Yi − M )
2
i =1
▪ There are easier computational formulas:

2
 k

  WiYi 
W Y 2 −  i =1 
k
Q= i i k
i =1
W i =1
i
▪ It is distributed as a chi-square with k-1 degrees of freedom,

where k is the number of ES
➢ Q has low power when the number of studies is low and when
sample size within studies is low
➢ Higgins and Thompson (2002) proposed I2:

Q − df
I = 100% 
2
Q
➢ Larger values of I2, the more heterogeneity
➢ 75% = considerable heterogeneity

➢ 50% = moderate heterogeneity
➢ 25% = low heterogeneity
The amount of
heterogeneity in the true
Fitting a REM (method = REML by default) log RR is estimated to be
0.3132
Considerable
heterogeneity
v The distribution of ES is
heterogeneous
Obtaining a forest plot: forest( )

Step 6. Moderators analyses
Heterogeneous distribution: What do we do then?
Analyse excess between study (ES) variability
continuous variables and/or

categorical variables multiple variables
analogous to one-way
weighted multiple regression –
ANOVA
Meta-regression
➢ Useful for a single categorical independent variable
➢ Test whether the grouping variable accounts for significant

variability in ES
➢ Produce a separate mean ES for each category (and also SE and

CI)
➢ Partition the total sum-of-squares (Q) into two pieces:

▪ Variability between groups (Qbetween) – analogous to an F-test between means
▪ Residual variability within groups (Qwithin) – assesses whether residual
distribution is homogeneous
Q = Qbetween + Qwithin
The Metafor package can test for moderatos

Argument to indicate the subset of studies that should be used for the analysis
Argument to specify moderators

Analogous
XXX to one-way ANOVA
dat.bcg – 13 studies of the effectiveness of the BCG vaccine against
tuberculosis (Colditz et al., 1994)
Moderator: type of allocation (alternate, random, systematic)
H 0 : β1 = β 2 = 0
The type of allocation method
v does not influence the average
effectiveness of the vaccine
β1= 0.4478 and β2 = 0.5369 estimate how much larger the average
Average log RR for random log RR values are when using alternate and systematic allocation,
allocation respectively.
Β1 + β1 = -0.9658 + 0.4478 = -0.5180 is the average log RR for
studies using alternate allocation
Β1 + β1 = -0.9658 + 0.5369 = -0.4289 is the average log RR for
studies using systematic allocation
◼ Often called meta-regression, since it is conceptually identical to multiple

regression
◼ ES is the dependent variable - Study moderator variables are the independent

variables
◼ Can handle multiple variables simultaneously (categorical and continuous

variables)
◼ Q for the model (QM) indicates whether the regression model explains a
significant portion of the variability across ES
◼ Q for the residual (QE) indicates whether the remaining variability across ES is
homogeneous
Q = QModel + QError
Meta-regression
dat.bcg – 13 studies of the effectiveness of the BCG vaccine against tuberculosis
(Colditz et al., 1994)
Moderators: year of publication (year), absolute latitude (ablat)
Tau-squared without moderators was
0.3132. So, (0.3132-0.1108)/0.3132 = 65%
v
of the total amount of heterogeneity can
be accounted for by including the two
moderators in the model
Test for residual heterogeneity is significant, indicating that other moderators

not considered in the model are influencing the vaccine effectiveness
H0: β1= β2 = 0
We can reject H0 but only absolute latitude appears to have a significant
influence on the effectiveness of the vaccine
One degree increase in absolute

latitude corresponds to a change of -
0.03 units in terms of the average
log RR (RR = 0.97). As we move
further away from the equator, the
average infection risk decreases
With the data: Example_ChallengePrograms
1. Check the data

4. Obtain the forest plot
5. What are the values of Q and I2? Are ES homogeneous?
6. Carry out a meta-regression with the two moderators: random
assignment to conditions and coder rated intensity of challenge
(from 1 = very low to 7 = very high)
Step 7. Exploring biases
➢ What research should be included or excluded in terms of

quality?
➢ Only high-quality studies?
➢ If both low- and high-quality studies, then sensitivity analysis?
➢ How do we measure methodological quality? There are several

instruments but there is no agreement as to what constitutes
research quality
➢ Statistically significant or larger effects are more likely to be

published than non-significant effects
➢ Reasons for it:

• journal policy for publishing results
• technical reports for government agencies, PhD theses may
not get published
• unknown language
• difficulty of access: search only known databases & journals,
only electronic files, only cost-free files
➢ Publication bias affects the validity of MA
➢ Search for and include unpublished studies that meet eligibility

criteria
➢ Compare results from published with unpublished studies

(sensitivity analysis)
➢ Assess distribution of effect sizes for publication bias
▪ Funnel plot: scatterplot of ES against SE

▪ Trim-and-fill method (Duvall & Tweedie, 2000)
From Rothstein et al. (2005)

▪ Visual tool for the study of publication and other bias

▪ Scatter plots of effects estimated from individual studies against a
measure of study size (usually SE)
▪ In the absence of bias the plot will resemble a symmetrical
inverted funnel
▪ Bias is evident when smaller studies showing no statistically
significant effects remain unpublished – funnel plot will appear
asymmetrical – the combined ES from MA will overestimate the
average effect or strength of the relationship

tuberculosis (Colditz et al., 1994)
▪ Non-parametric (rank-based) data augmentation technique

proposed by Duval and Tweedie (2000)
▪ It adjusts MA for the impact of missing studies
▪ Analysis of the potential effect that missing studies may have had
on the observed result
▪ It can be used to estimate the number of studies missing from
MA due to the suppression of the most extreme results on one
side of the funnel plot
▪ Steps:
1. Eliminate iteratively the most extreme small-studies’ ES until
symmetry is reached.
2. Estimate new ES.
3. Put eliminated back + impute until symmetry: this step corrects
against underestimating the average ES variance, but does not affect
the estimate of the mean.

tuberculosis (Colditz et al., 1994) - FEM
The estimated effect of the vaccine is smaller with the missing studies filled
in, but results still indicate that the effect is statistically significant
With the data: Example_TherapyDepression
1. Check the data

4. Obtain the forest plot
5. What are the values of Q and I2? Are ES homogeneous?
6. Explore the effect of sample (in-patient/out-patient) on ES
7. Obtain the funnel plot
8. Apply the trim and fill method
9. Is there any evidence of publication bias?

Step 8. Interpreting and presenting
➢ If results are consistent – the focus of the report is likely to be on

the summary effect
➢ If ES vary modestly from study to study – summary effect +

dispersion in effects
➢ If ES vary substantially – focus on the variance + summary effect

less (or not) important
Guidelines for reporting MA
➢ MOOSE reporting meta-analysis of observational studies in

epidemiology (Stroup et al., 2000)
➢ PRISMA statement for reporting systematic reviews and meta-

analysis (Liberati et al., 2009)
➢ Cochrane Handbook for Systematic Reviews of Interventions

(Higgins and Green, 2008)
1. Structural Equation Modelling (SEM) meta-analysis (software:
metaSEM package in R)
2. Network meta-analysis (software: WinBUGS, netmeta package in R)
3. Individual-Participant-Data meta-analysis
4. Meta-analysis of single-case designs
5. 3D-data meta-analysis (ALE method, software: GingerALE and Matlab)
6. Psychometric meta-analysis (Hunter and Schmidt approach)

In summary, the goals of MA are to:
1. Describe the distribution of ES, including its mean

2. Establish a CI around the mean ES
3. Test that the mean ES differs from 0
4. Test whether studies tell a consistent story (are
homogeneous)
5. Explore the relationship between study features and ES
From Wilson (2010)

➢ Meta-analysis is a reproducible and defensible method of
summarizing findings across studies
➢ Meta-analysis often points out gaps in the research

literature, providing a solid foundation for the next
generation of research on that topic
➢ Meta-analysis illustrates the importance of replication
➢ Meta-analysis facilitates generalization of the knowledge

gained through individual evaluations
From Wilson (2010)

If you are going to carry out MA, I strongly recommend:
▪As an initial reading, have a look at the book Practical Meta-analysis by
Lipsey and Wilson (2001)
▪Read some empirical MA from the Psychological Bulletin (the Hedges

and Olkin approach) and the Journal of Applied Psychology (the Hunter
and Schmidt approach)
▪Follow publications from Wolfgang Viechtbauer (empirical and

methodological studies) and his updates of the Metafor package
▪If you are going to use R, read the book by Chen and Peace (2013)
Applied meta-analysis with R (Chapman & Hall/CRC Biostatistics
Series)
Web pages of interest:
▪ Cochrane Collaboration: www.Cochrane.org
▪ Campbell Collaboration: www.campbellcollaboration.org
▪ David Wilson’s personal web page
▪ Wolfgang Viechtbauer’s personal web page

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley &
Sons.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale. NJ: Erlbaum.
Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Sage Publications.
Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication
bias in meta‐analysis. Biometrics, 56(2), 455-463.
Higgins, J. P. (Ed.). (2008). Cochrane handbook for systematic reviews of interventions (Vol. 5). Chichester: Wiley-
Blackwell.
Higgins, J., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. Bmj,
327(7414), 557-560.
Guilera, G., Barrios, M., & Gómez-Benito, J. (2012). Meta-analysis in psychology: a bibliometric study. Scientometrics,
94(3), 943-954.
Guilera, G., Pino, O., Gómez-Benito, J., & Rojo, J. E. (2009). Antipsychotic effects on cognition in schizophrenia: A
meta-analysis of randomised controlled trials. European journal of psychiatry, 23(2), 77-89.
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., ... & Moher, D. (2009). The PRISMA
statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions:
explanation and elaboration. PLoS Medicine, 6(7): e1000100. doi: 10.1371/journal.pmed.1000100.
Lipsey, M. W., & Wilson, D. (2000). Practical meta-analysis (applied social research methods). Thousand Oaks, CA:
Sage Publications.
Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford: Oxford University Press.
Purdon, S. E., Malla, A., Labelle, A., & Lit, W. (2001). Neuropsychological change in patients with schizophrenia after
treatment with quetiapine or haloperidol. Journal of Psychiatry and Neuroscience, 26(2), 137.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2006). Publication bias in meta-analysis: Prevention,
assessment and adjustments. John Wiley & Sons.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software,
36(3), 1-48.
May all your effects be positive!
Georgina Guilera – gguilera@ub.edu

Meta Analysis 2019 2020

Uploaded by

Copyright:

Available Formats

You might also like

Meta Analysis 2019 2020

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Meta Analysis 2019 2020

Uploaded by

Copyright:

Available Formats

M. Sc.

Research in Behavior &

2. Systematic review vs. meta-analysis

3. Definition and objectives of meta-analysis

➢ Science is a cumulative process

➢ Many studies address the same basic question

➢ Researchers try to summarize the literature on a particular

➢ There are several strategies, e.g., narrative review, scoping

➢ 1952 --- Hans J. Eysenck concluded that there were no favourable

➢ After him… Olkin, Cohen, Rosenthal, Rubin, Hunter, Schmidt,

From Guilera et al. (2012)

➢ Systematic review is a revision of the literature that requires

➢ Meta-analysis is a systematic literature review supported by

Meta-analysis = Systematic review + quantitative estimation of effects

The streptokinase example

➢ Streptokinase was supposed to increase the likelihood of survival

➢ All the trials followed similar protocols (random assignment to treatment or

➢ 6 studies were statistically significant in favour to streptokinase while the

➢ Meta-analysis is a systematic literature review supported by

➢ Offer a guide to practitioners regarding intervention

relevant moderator variables

➢ Careful statement of the topic to be studied or the

➢ This statement will guide the selection of studies, the

➢ The statement should be complete, but initially need

➢ Identify the form(s) of research findings relevant to the topic of

How effective was a treatment in the

➢ Significant findings are more likely to be published than

➢ Searching far and wide – Grey literature – Quality of

▪ Computerized bibliographic databases – which keywords?

➢ Identifying inclusion/exclusion criteria:

➢ Difficult to retrieve all studies that meet your eligibility

➢ Excel is a good option, but there are specific tools

➢ Key point: Clearly explain your search strategy,

Image extracted from the APA Style Blog:

➢ Think about a topic you are interested in that would be

➢ Try to select your keywords, define your search strategy, choose

➢ Search in one of the databases

➢ How many reports did you find?

➢ How the results are reported

➢ Characteristics of the setting and subjects

➢ Data on treatments used (if relevant)

➢ Study outcomes Potential

➢ Development of coding protocol (manual + form)

➢ Assessing the accuracy of extracted information

From Lipsey and Wilson (2001)

From Lipsey and Wilson (2001)

 First session: coding manual and form (paper,

 Regular meetings - develop normative understanding

 Write comments on the coding manual

 Be trained (manual + form)

 Several constructs – e.g., treatment effects on social skills,

➢ Example SGA vs Haloperidol

From Guilera et al. (2009)

 Select one via a substantive criterion

 Sample one at random

➢ Consistency (≥ 20% documents)

 Intra-rater reliability – consistency of a single coder

 Inter-rater reliability (double coding) – consistency