RETRIEVING AND CRITICAL APPRAISAL

STUDI FARMAKOEKONOMI
Dwi Endarti, MSc, PhD, Apt
Lab Manajemen Farmasi dan Farmasi Masyarakat, Departemen Farmasetika
Fakultas Farmasi UGM
How to get PE/EE studies
• Doing PE/EE studies:
• Need more knowledge and longer time than
retrieving PE/EE studies

•Retrieving both international and national

studies:
• Easy and convenient
• Best way to start
2
Type of information
• Conference and presentation proceeding
• Reports
• Dissertation and thesis (grey literature → may not be published in the
journal)
• Journal publication
• Textbooks

→ all type of information could be :

• Paper base
• Electronic base

3
Finding EE studies
• Own knowledge
• Contact experts in the filed
• Email discussion lists
• Hand search of publications (journals, books, reports) and grey
literature
• General or specialist electronic databases: international/national

4
International electronic databases
• General bibliographic databases
• Medical research databases:
• Pubmed,
• Embase,
• Cochrane
• Website of international journals:
• BMJ,
• Lancet,
• JAMA,
• Health Policy and Planning,
• AJPH,
• Plos Medicine

5
International electronic databases
• Specialist pharmacoeconomics databases
• Website of HTA organization:
• CRD-NIHR HTA programme (Centre for Reviews and Dissemination-National Institute for
Health Research)
• PATH (Program for Appropriate Technology in Health)
• NICE (National Institute for Health and Clinical Excellent)
• INAHTA (International Agency of Health Technology Assessment)
• HERC (Health Evidence Review Commission)
• CEA registry (Cost Effectiveness Analysis registry)

10
International electronic databases
• Specialist pharmacoeconomics databases
• Website of international specialist pharmacoeconomics databases:
• Cost-Effectiveness and Resource Allocation
• Value in Health
• Pharmacoeconomics
• European Journal of Health Economics
• Health Economics
• International Journal Technology Assessment in Healthcare
• Medical Decision Making

12
Retrieving from other sources
• To disseminate and exchange HTA knowledge between expert and
interested persons:
• Conference and training
• Disseminate knowledge and information (book, newsletter, journal, email etc)
• Job announcement
• Organization such as:
• IHEA (International Health Economics Association)
• ISPOR (International Society for Pharmacoeconomics and Outcome Research)

15
Objective of critical assessment of EE studies
• To select EE study correctly
• Many studies seeming to be EE studies are a not
• These studies may not be used for policy decision making related to cost-
effectiveness
→ Screening for EE studies
• To get EE studies with high-quality
→comparing with standard criteria for reporting EE
• To evaluate the reliability of EE results
• Many studies do not follow clear methods
• Possibility for manipulation of results
• Finally, EE studies can be used as the information for policy making : drug
registration, NLED selection, hospital formulary selection

17
Screening for EE studies
Is there a comparison of two or
more alternatives?
The study is not an health-
yes No economic evalutaion;
But it may be useful
Are both costs and health
benefits (dis-benefits)
examined?

yes

Economic evaluation study and

need to be critically appraisal

18
Critical appraisal of EE studies
• What is to be appraised?
• Methodology used
• Evidence used (input data)
• Reporting
→Using guideline
• Assessment of methodological used should be presented as:
• Checklist
• Describe and discuss evaluation results based on the appropriate guidelines
• Explain method, results, strength, and weakness which has impact on the
reliability of conclusion

19
 Some Guidelines for critical appraisal of EE study
• 10-point checklist for EE
• (Drummond et al. Methods for the economic
evaluation of health care programmes, 2005)
• CHEERS
• (Husereau, Don, et al. "Consolidated health economic
evaluation reporting standards (CHEERS) statement."
BMC medicine 11.1 (2013): 80)
• Husereau, Don, et al. "Consolidated Health Economic
Evaluation Reporting Standards 2022 (CHEERS 2022)
statement: updated reporting guidance for health
economic evaluations." Health Policy OPEN (2022):
100063.
• ECOBIAS
• (Adarkwah CC, van Gils PF, Hiligsmann M, Evers SM.
Risk of bias in model-based economic evaluations: the
ECOBIAS checklist. Expert review of
pharmacoeconomics & outcomes research. 2016 Jul
3;16(4):513-23)
• Transferability
• Welte R, Feenstra T, Jager H, Leidl R. A decision chart
for assessing and improving the transferability of
economic evaluation results between countries.
Pharmacoeconomics. 2004 Sep;22(13):857-76.

20
10-point of ‘Drummond’ checklist
1.Was a well-defined question posed in answerable form?
2.Was a comprehensive description of alternatives given?
3.Was there evidence that effectiveness had been established?
4.Were all the important and relevant costs and consequences for each alternative
identified?
5.Were costs and consequences measured accurately/appropriately?
6.Were costs and consequences valued credibly?
7.Were costs and consequences adjusted for differential timing?
8.Was an incremental analysis performed?
9.Was allowance made for uncertainty?
10.Did presentation/discussion of results include all issues of concern?

21
Example of critical appraisal of published EE article using Drummond
checklist

22
Guideline of Drummond et al: checklist 1

• Was a well-defined question posed in answerable form?

• Did the study examine both costs and effects of the service(s) or programme(s)?
• Did the study involve a comparison of alternatives?
• Was a viewpoint for the analysis stated and was the study placed in any particular
decision-making context?

23
Guideline of Drummond et al: checklist 2

• Was a comprehensive
description of the competing
alternatives given (i.e. can you
tell who did what to whom,
where, and how often)?
• Were there any important
alternatives omitted?
• Was (should) a do-nothing
alternative be considered?

24
Guideline of Drummond et al: checklist 3

• Was the effectiveness of the programme or services

established?
• Was this done through a randomised, controlled clinical trial? If so, did
the trial protocol reflect what would happen in regular practice?
• Was effectiveness established through an overview of clinical studies?
• Were observational data or assumptions used to establish
effectiveness? If so, what are the potential biases in results?

25
Guideline of Drummond et al: checklist 4

• Were all the important and

relevant costs and consequences
for each alternative identified?
• Was the range wide enough for the
research question at hand?
• Did it cover all relevant viewpoints?
(Possible viewpoints include the
community or social viewpoint, and
those of patients and third-party
payers. Other viewpoints may also be
relevant depending upon the
particular analysis.)
• Were the capital costs, as well as
operating costs, included?

26
Guideline of Drummond et al: checklist 5
• Were costs and consequences measured accurately in
appropriate physical units (e.g. hours of nursing time,
number of physician visits, lost work-days, gained life
years)?
• Were any of the identified items omitted from measurement? If
so, does this mean that they carried no weight in the subsequent
analysis?
• Were there any special circumstances (e.g., joint use of
resources) that made measurement difficult? Were these
circumstances handled appropriately?
• Example: cost of drug = number of tablet x price per
tablet (the data is not provided in the article)

27
Guideline of Drummond et al: checklist 6
• Were the cost and consequences valued credibly?
• Were the sources of all values clearly identified? (Possible
sources include market values, patient or client preferences
and views, policy-makers’ views and health professionals’
judgements)
• Were market values employed for changes involving
resources gained or depleted?
• Where market values were absent (e.g. volunteer labour), or
market values did not reflect actual values (such as clinic
space donated at a reduced rate), were adjustments made to
approximate market values?
• Was the valuation of consequences appropriate for the
question posed (i.e. has the appropriate type or types of
analysis – cost-effectiveness, cost-benefit, cost-utility – been
selected)?

28
Guideline of Drummond et al: checklist 7
• Were costs and consequences adjusted for
differential timing?
• Were costs and consequences that occur in the future
‘discounted’ to their present values?
• Was there any justification given for the discount rate
used?

29
Guideline of Drummond et al: checklist 8
• Was an incremental
analysis of costs and
consequences of
alternatives performed?
• Were the additional
(incremental) costs
generated by one
alternative over another
compared to the additional
effects, benefits, or utilities
generated?

30
Guideline of Drummond et al: checklist 9
• Was allowance made for
uncertainty in the estimates of
costs and consequences?
• If data on costs and consequences
were stochastic (randomly
determined sequence of
observations), were appropriate
statistical analyses performed?
• If a sensitivity analysis was
employed, was justification
provided for the range of values (or
for key study parameters)?
• Were the study results sensitive to
changes in the values (within the
assumed range for sensitivity
analysis, or within the confidence
interval around the ratio of costs to
consequences)?

31
Guideline of Drummond et al: checklist 10
• Did the presentation and discussion of study
results include all issues of concern to users?
• Were the conclusions of the analysis based on
some overall index or ratio of costs to
consequences (e.g. cost-effectiveness ratio)? If
so, was the index interpreted intelligently or in
a mechanistic fashion?
• Were the results compared with those of
others who have investigated the same
question? If so, were allowances made for
potential differences in study methodology?
• Did the study discuss the generalisability of the
results to other settings and patient/client
groups?
• Did the study allude to, or take account of,
other important factors in the choice or
decision under consideration (e.g. distribution
of costs and consequences, or relevant ethical
issues)?
• Did the study discuss issues of
implementation, such as the feasibility of
adopting the ‘preferred’ programme given
existing financial or other constraints, and
whether any freed resources could be
redeployed to other worthwhile programmes?

32
Example of application EE critical appraisal in systematic review
article

33
Example of application EE critical appraisal in systematic review
article

34
Example results of systematic review of EE
studies

35
36
CHEERS checklist - 2013

37
CHEERS – checklist 2022

38
ECOBIAS Checklist

39
Transferability – Checklist

40
TERIMA KASIH