Evidence Synthesis

Introduction to Systematic Review and Meta-Analysis

Septiara Putri
Health Policy and Administration Department, FPH UI
Center for Health Economics and Policy Studies
2018
1
 Outline
• Introduction to systematic review (epidemiological
studies)
• How to conduct systematic review
• Meta-analysis: the basics
• Critical appraisal/assessment
• Interpretation and reporting
• Exercise

2
 Definition
• “A systematic review attempts to collate all empirical
evidence that fits pre-specified eligibility criteria in order
to answer a specific research question”

• “Systematic methods that are selected with a view to

minimizing bias, thus providing more reliable findings
from which conclusions can be drawn and decisions
made”
Higgins, J. P. T. (2014). Green S. Cochrane handbook for systematic reviews of interventions Version 5.1. 0 [updated March 2011].
The Cochrane Collaboration, 2011. www. cochrane-handbook. org.

3
 Hierarchy of Evidence

Figure 1. Hierarchy of Evidence

4
GRADE approach
The Grades of Recommendation,
Assessment, Development and
Evaluation

“ For purposes of systematic

reviews, the GRADE approach
defines the quality of a body of
evidence as the extent to which
one can be confident that an
estimate of effect or association is
close to the quantity of specific
interest. “

http://handbook.cochrane.org/chapter_12/12_2_1_the_grade_approach.htm
5
 Narrative vs Systematic Review

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Systematic reviews: synthesis of best evidence for clinical decisions. Annals of internal medicine,126(5), 376-380

6
 Key Characteristics
 a clearly stated set of objectives with pre-defined eligibility criteria for
studies;

 an explicit, reproducible methodology;

 a systematic search that attempts to identify all studies that would

meet the eligibility criteria;

 an assessment of the validity of the findings of the included studies,

for example through the assessment of risk of bias; and

 a systematic presentation, and synthesis, of the characteristics and

findings of the included studies. 7
 What are systematic reviews needed?
 Specialization in health field, increasing number of
publication
 Internet access to large databases
 Good quality of evidence
 Careful strategies and evidence for decision making
 The failing of narrative review

8
 High quality systematic review
High quality systematic reviews seek to
• Identify all relevant published and unpublished evidence

• Select studies or reports for inclusion

• Assess the quality of each study or report

• Synthesise the findings from individual studies or reports in an unbiased way

• Interpret the findings and present a balanced and impartial summary of the findings
with due consideration of any flaws in the evidence.
What is Systematic Review? http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/syst-review.pdf

10
Databases and Guideline

SP| Economic Evaluation | Session 7 10

 Steps conducting systematic review

1. Formulate the review question

2. Define inclusion and exclusion criteria
3. Develop search strategy and locate studies
4. Select studies
5. Extract data
6. Assess study quality
7. Analyze and interpret results
8. Disseminate findings
SP| Economic Evaluation | Session 7 11
 Figure 2. Systematic review flow

Explore the part of Cochrane Systematic Review http://navigatingeffectivetreatments.org.au/exploring_systematic_reviews.html

SP| Economic Evaluation | Session 7 12
 1. Formulate the review question
• “The first stage involves defining the review
question, forming hypotheses, and developing a
review title. It is often best to keep titles as short
and descriptive as possible, by using the following
formula: Intervention for population with condition”

Example: Systematic review of community-based

childhood
obesity prevention studies by Bleich et al.,2013
Uman, L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20(1), 57.
Bleich, S. N., Segal, J., Wu, Y., Wilson, R., & Wang, Y. (2013). Systematic review of community-based childhood obesity prevention
studies. Pediatrics,132(1), e201-e210

SP| Economic Evaluation | Session 7 13

2. Define inclusion and exclusion
criteria
Define PICOS
•Population: children and adolescents aged 2 to 18 years old,
high income countries
•Intervention: a modification of diet, a modification of physical
activity or sedentary activity, or a combination of these
•Comparators: usual care, another different intervention, or
no intervention
•Outcomes: the attained differences between the intervention
and control groups in weight-related outcomes
•Study designs: randomized controlled trials (RCTs), quasi-
experimental studies, and natural experiment.

SP| Economic Evaluation | Session 7 14

 2. Define inclusion and exclusion
criteria
• “Studies were excluded if they were (1) observational (eg, cross-sectional); (2)
targeted only at overweight or obese children or adolescents; (3) targeted only at
children or adolescents with chronic medical condition (eg, diabetes or heart
disease); (4) expressly targeted at weight loss; (5) collected only qualitative results
(eg, interviews or focus groups); (6) or published only in abstract form. We
differentiated natural experiments from other observational study designs by
specifying that a natural experiment was the implementation of a policy or similar
intervention at a population level.”

SP| Economic Evaluation | Session 7 15

3. Develop search strategy and locate
studies

- List comprehensive of terms [MeSH], possible

synonym
- Perform syntax if needed “*” or “$”
- Assign “OR” and “AND”
- Be clear with explode, focus tree, limits, years
- Librarian could help!

SP| Economic Evaluation | Session 7 16

SP| Economic Evaluation | Session 7 17
 Boolean operator

http://guides.lib.umich.edu/c.php?g=283340&p=2126706

18
Search strategy example (1)

20
 Search strategy example (2)

Carrier, M., Le Gal, G., Wells, P. S., & Rodger, M. A. (2010). Systematic review: case-fatality rates of recurrent venous thromboembolism and
major bleeding events among patients treated for venous thromboembolism. Annals of internal medicine, 152(9), 578-589.

20
 4. Select studies

• “ Once a comprehensive list of abstracts has been

retrieved and reviewed, any studies appearing to
meet inclusion criteria would then be obtained and
reviewed in full. This process of review is generally
done by at least two reviewers to establish inter-
rater reliability”

SP| Economic Evaluation | Session 7 21

 5. Extract data

• “It can be helpful to create and use a simple data

extraction form or table to organize the information
extracted from each reviewed study (e.g., authors,
publication year, number of participants, age range,
study design, outcomes, included/excluded). Data
extraction by at least two reviewers is important
again for establishing inter-rater reliability and
avoiding data entry errors.”

SP| Economic Evaluation | Session 7 22

 6. Assess study quality

• Is the topic well defined

• Was the search for papers thorough?
• Were the criteria for inclusion of studies clearly described and
fairly applied?
• Was study quality assessed by blinded or independent
reviewers?
• Was missing information sought from the original study
investigators?
• Do the included studies seem to indicate similar effects?
• Were the overall findings assessed for their robustness
• Was the play of chance assessed?
• Are the recommendations based firmly on the quality of the
evidence presented?
SP| Economic Evaluation | Session 7 23
Li, Y., Liu, L., Wang, B., Wang, J., & Chen, D. (2013). Metformin in non-alcoholic fatty liver disease: A systematic review and
meta-analysis.Biomedical reports, 1(1), 57-64.

SP| Economic Evaluation | Session 7 24

 Critical appraisal guideline
• AMSTAR: assessing methodological quality of systematic
reviews
• 5-point Oxford Quality Rating Scale
• Consolidated Standards of Reporting Trials (CONSORT)
• Critical Appraisal Skills Programme (CASP)
• Critical Appraisal Tools By CEBM
• Appraisal of a systematic review using a critical appraisal ch
ecklist
BMJ
• Critical Appraisal and Data Extraction for Systematic Reviews
of Prediction Modelling Studies: The CHARMS Checklist
SP| Economic Evaluation | Session 7 25
• Cochrane Risk of Bias tool for randomised therapeutic
studies, QUADAS (and QUADAS-2) for diagnostic
accuracy studies, and the QUIPS checklist for appraisal
of prognostic factor studies

• PRISMA , Preferred Reporting Items for Systematic

Reviews and Meta-Analyses

• The Grades of Recommendation, Assessment,

Development and Evaluation Working Group (GRADE
Working Group)

SP| Economic Evaluation | Session 7 26

7. Analyze and interpret results
• Flow that you conduct systematic review
• Need for meta-analysis?
• Publication bias?
• Limitation of study?
• Generalizability
• Comparison with another studies?

8. Disseminate findings
SP| Economic Evaluation | Session 7 27
Presenting Results

SP| Economic Evaluation | Session 7 28

 META-ANALYSIS
• Meta-analysis is a statistical technique
for
combining the findings from independent studies.

• Meta-analysis is most often used to assess the

clinical effectiveness of healthcare interventions; it
does this by combining data from two or more
randomised control trials.
• http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/meta-an.pdf

SP| Economic Evaluation | Session 7 30

 Fixed Effect vs Random Effect
Under the fixed effect model we assume that there is
one true effect size which is shared by all the included
studies. It follows that the combined effect is our
estimate of this common effect size.

Under the random effects model we allow that the true

effect could vary from study to study.

https://www.meta-analysis.com/downloads/Meta-analysis%20fixed%20effect%20vs%20random%20effects.pdf

SP| Economic Evaluation | Session 7 30

Lohse, K., 2013

SP| Economic Evaluation | Session 7 31

 Fixed effect

Lohse, K., 2013

SP| Economic Evaluation | Session 7 32

Lohse, K., 2013

SP| Economic Evaluation | Session 7 33

Lohse, K., 2013

SP| Economic Evaluation | Session 7 34

SP| Economic Evaluation | Session 7 35
 How to read meta-analysis

Li, Y., Liu, L., Wang, B., Wang, J., & Chen, D. (2013). Metformin in non-alcoholic fatty liver disease: A systematic review and meta-analysis.Biomedical reports, 1(1),
57-64.
Risk ratios and 95% CIs for fully adjusted random-effects models examining associations between saturated fat intake in relation to
coronary heart disease mortality. ¹Studies that included adjustments for serum or LDL cholesterol. SAT, saturated fat intake.

SP| Economic Evaluation | Session 7 36

Te Morenga, L., Mallard, S., & Mann, J. (2013). Dietary sugars and body weight: systematic review and meta-analyses of randomised
controlled trials and cohort studies. Bmj, 346, e7492.

SP| Economic Evaluation | Session 7 37

 Systematic Review in Economic
Evaluation
• 10 main critical questions by Drummond et al., 2005.
• Assessment/critical appraisal not all studies perfectly
perfect or perfectly rubbish.
• Exercise critical appraisal economic evaluation studies (for
this week)

Graham, D. M., Isaranuwatchai, W., Habbous, S., de

Oliveira, C., Liu, G., Siu, L. L., & Hoch, J. S. (2014). A
cost‐effectiveness analysis of human papillomavirus
vaccination of boys for the prevention of
oropharyngeal cancer.Cancer.

SP| Economic Evaluation | Session 7 38

1. Was a well-defined question posed in
answerable form?

1.1. Did the study examine both costs and effects

of the service(s) or programme(s)?
2. Did the study involve a comparison of
alternatives
3. Was a viewpont for the analysis stated and was
the study placed in any particular decision
making context?
SP| Economic Evaluation | Session 7 40
2. Was a comprehensive description of the
competing alternatives given? (that is can you
tell who did what to whom, where and
how
often?)

1. Were anly relevant alternatives omitted?

2. Was (should) a “do-nothing”
alternative be considered?

SP| Economic Evaluation | Session 7 40

3.Was the effectiveness of the programme or
services established?
1.Was this done through randomized, controlled
clinical trial? If so, did the trial protocol reflect what
would happen in regular practice?
2.Were effectiveness data collected and
summarised through a systematic overview of
clinical studies?
3.Were observational data or assumptions used to
establish effectiveness? If so, what are the potential
biases of the results?
SP| Economic Evaluation | Session 7 41
4.Were all the important and relevant costs and
consequences of each alternative identified?

1.Was the range wide enough for the research

question at hand?
2.Did it cover all relevant viewpoints?
3.Were capital costs, as well as operating costs,
included?

SP| Economic Evaluation | Session 7 42

5.Were costs and consequences measured accurately in
appropriate physical units? (for example, hours per nursing
time, nuber of physician visit, work days etc

1.Were the sources of resource utilization described and

justified?
2.Were any of identified items omitted from measurement? If
so, does this mean that they carried no weight in the
subsequent analysis?
3.Were there any special circumstances (for example, joint
use of resources) that made measurement difficult? Were
there circumtances handled appropriately

SP| Economic Evaluation | Session 7 43

6. Were costs and consequences valued credibly?
1.Were the sources of all values clearly identified?
2.Were market values employed for changes
involving resources gained or depleted?
3.Where market values were absent, were
adjustments made to approximate market values?
4.Was the valuationof consequences
appropriate for the question pose?

SP| Economic Evaluation | Session 7 44

7.Were costs and consequences adjusted for
differential timing?

1.Were costs and consequences that occur in the

future ‘discounted’ to their present values?
2.Was any justification given for the discount rate
used?

SP| Economic Evaluation | Session 7 45

8. Was incremental analysis of costs and
consequences of alternatives performed?

8.1 Was an incremental analysis of costs

and consequences of alternatives
performed?

SP| Economic Evaluation | Session 7 46

 9. Was allowance made for uncertainty in
the estimates of costs and consequences?

1.If patient level data on costs or consequences were

available, were appropriate statistical analysis performed?
2.If a sensitivity analysis was employed, was justification
provided for the ranges or distributions of values, and the form
sensitivity analysis used?
3.Were the conclusions of the study sensitive to the
uncertainty in the result, as quantified by the statistical and/or
sensitivity analysis

SP| Economic Evaluation | Session 7 47

10. Did the presentation and discussion of study
results include all issues of concern to users?

10.1 Were the conclusions of the analysis based on

some overall index or ratio of costs to consequences
(i.e: cost effectiveness ratio)? If so, was the index
interpreted intellegently or in mechanistic fashion?

10.2 Were the results compared with those of others

who have investigated the same question? If so, were
allowance made for potential differences in
study
methodology? SP| Economic Evaluation | Session 7
49
3.Did the study discuss the generalizability of the
results to other settings and patient/client groups?

4.Did the study allude to, or take account of, other

important factors in the choice or decision under
consideration (for example, distribution of costs and
consequences, or relevant ethical issues)?

SP| Economic Evaluation | Session 7 49

• 10.5 Did the study discuss issues of
implementation, such as the feasibility of adopting
the ‘preferred’ programme given existing financial
or other constraints, and whether any freed
resources could be redeployed to other worthwhile
programmes?

SP| Economic Evaluation | Session 7 50

Reporting standard

SP| Economic Evaluation | Session 7 51

 THANK YOU

septiara.putri@cheps.or.id

SP| Economic Evaluation | Session 7 52