D2L3 - Critical Appraisal Ec Ev v1

School of
Pharmaceutical
Sciences
Discipline of Social
& Administrative
Pharmacy
REPORTING & APPRAISAL OF

PHARMACOECONOMICS
EVALUATION STUDY
Assoc Prof Dr Asrul Akmal Shafie

BPharm, PG Dip Health Econs, PhD
aakmal@usm.my
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
A Growing Field
Wagstaff & Culyer (2012) J Health Econs

School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
A Growing Field
EARLY DAYS SINCE ‘80s
– standard economic used e.g. – Era of EBM
supply and demand--because – health economic evaluation.
most health economists were
trained in economics and just
– Multidisciplinary presented in a
happened to be interested in the framework that has been
market forces driving the health adapted from basic economics.
system. – Pharmaceutical companies,
– Traditionally trained economist Formulary-making bodies,
– inform government policies government agencies
largely dealing with
reimbursement for health care
services or drugs, and other
policy issues.
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
A Growing Questions
Drummond (1993) Int J Health Tech Assessment

School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
Malaysia Pec Studies
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
Malaysia Pec Studies
Recommendations Number of studies Percentage ( %)
fulfilling
recommendation
Perspective specified 7 63.6%
Used discounting for 10 90.9%
costs or/and outcomes if
study period was > 1 year
Calculated and reported 1 9.09%
ICER
Performed uncertainty 5 45.5%
analysis
Disclosed funding 6 54.5%
sources
School of
Importance of
Pharmaceutical
Sciences
& Administrative
Pharmacy
Standardization
• Meaningful comparison require standardization
– Without standardization, no assurance about the
comparability to results of different set of intervention.
– Method must be valid and applied consistently.
– Based on theory, can help ensure the standard to be
logically consistent, valid and credible.
• Maintain credibility in wider research community,
policy makers and health care professionals
School of
Importance of
Pharmaceutical
Sciences
& Administrative
Pharmacy
Standardization
• Provide protocols so those inexperienced in the
methodology of economic evaluation or the specific
need of the local authority can follow.
• Sponsors need to be sure that the analysts they
commission are employing good methods without being
unnecessarily perfectionist in their approach, with
resulting consequences for the cost and time that they
require for evaluations. Conversely, explicit statements
can help the analyst defend his or her position in
debates with a sponsor who wishes to cut corners in an
evaluation or to adopt potentially biased positions.
School of
Pharmaceutical
Sciences
Need to Standardize How to
& Administrative
Pharmacy
Report the Process?
School of
Pharmaceutical
Sciences
Increasing Pharmacoeconomic
& Administrative
Pharmacy
Standardization
• European Community Concerted Action on the Harmonization of the

Methodology for Economic Evaluation of Health Technology (HARMET)
• Panel on Cost-Effectiveness in Health and Medicine, US Department of Health
and Human Services
• International Society for Pharmacoeconomics and Outcomes Research
CHEERS Checklist
A brief list of recommendations subdivided into the five

sections generally found in an economic evaluation paper:
– Title and Abstract
– Introduction
– Methods
– Results
– Discussion Checklist is built on guidelines
and should be used together.
CHEERS Checklist – Items to include when reporting economic
evaluations of health interventions (1)
Item
Section/Item Recommendation
No
Title and abstract
Identify the study as an economic evaluation, or use more specific terms
Title 1 such as ``cost-effectiveness analysis``, and describe the interventions
compared.
Provide a structured summary of objectives, perspective, setting, methods
Abstract 2 (including study design and inputs), results (including base case and
uncertainty analyses), and conclusions.
Introduction
3 Provide an explicit statement of the broader context for the study.

Background and
objectives Present the study question and its relevance for health policy or practice
decisions.
Methods
Target Population and Describe characteristics of the base case population and subgroups
4
Subgroups analyzed including why they were chosen.
State relevant aspects of the system(s) in which the decision(s)
Setting and Location 5
need(s) to be made.
Describe the perspective of the study and relate this to the costs
Study Perspective 6
being evaluated.
Describe the interventions or strategies being compared and state
Comparators 7
why they were chosen.
State the time horizon(s) over which costs and consequences are
Time Horizon 8
being evaluated and say why appropriate.
Item
No
Report the choice of discount rate(s) used for costs and outcomes and
Discount Rate 9
say why appropriate.
Choice of Health Describe what outcomes were used as the measure(s) of benefit in the
10
Outcomes evaluation and their relevance for the type of analysis performed.
Single Study-Based Estimates: Describe fully the design features of
11a the single effectiveness study and why the single study was a sufficient
Measurement of source of clinical effectiveness data.
Effectiveness
Synthesis-based Estimates: Describe fully the methods used for identi-
11b fication of included studies and synthesis of clinical effectiveness data.
Measurement and
Valuation of If applicable, describe the population and methods used to elicit preferences
12
Preference-Based for outcomes.
Outcomes
Single Study-based Economic evaluation: Describe approaches used to
estimate resource use associated with the alternative interventions. Describe
13a primary or secondary research methods for valuing each resource item in
terms of its unit cost. Describe any adjustments made to approximate to
Estimating opportunity costs.
Resources and Costs Model-based Economic Evaluation: Describe approaches and data sources
used to estimate resource use associated with model health states. Describe
13b primary or secondary research methods for valuing each resource item in
terms of its unit cost. Describe any adjustments made to approximate to
opportunity costs.
Item
No
Report the dates of the estimated resource quantities and unit costs.
Currency, Price Date Describe methods for adjusting estimated unit costs to the year of reported
14
and Conversion costs if necessary. Describe methods for converting costs into a common
currency base and the exchange rate.
Describe and give reasons for the specific type of decision-analytic model
Choice of model 15
used. Providing a figure to show model structure is strongly recommended.
Describe all structural or other assumptions underpinning the decision-
Assumptions 16
analytic model.
Describe all analytic methods supporting the evaluation. This could include
methods for dealing with skewed, missing or censored data, extrapolation
Analytic Methods 17 methods, methods for pooling data, approaches to validate or make
adjustments (e.g., half-cycle corrections) to a model, and methods for
handling population heterogeneity and uncertainty.
Results
Report the values, ranges, references and if used, probability distributions for
all parameters. Report reasons or sources for distributions used to represent
Study parameters 18
uncertainty where appropriate. Providing a table to show the input values is
strongly recommended.
For each intervention, report mean values for the main categories of
Incremental costs estimated costs and outcomes of interest, as well as mean differences
19
and outcomes between the comparator groups. If applicable, report incremental cost-
effectiveness ratios.
Item
No
Single study-based economic evaluation: Describe the effects of sampling
uncertainty for estimated incremental cost, incremental effectiveness and
20a
incremental cost-effectiveness, together with the impact of methodological
Characterizing
assumptions (e.g. discount rate, study perspective).
Uncertainty
Model-based economic evaluation: Describe the effects on the results
20b of uncertainty for all input parameters, and uncertainty related to the
structure of the model and assumptions.
If applicable, report differences in costs, outcomes or cost-
effectiveness that can be explained by variations between subgroups
Characterizing 21 of patients with different baseline characteristics or other observed
Heterogeneity variability in effects that are not reducible by more information.
Discussion
Study Findings,
Summarize key study findings and describe how they support the
Limitations,
22 conclusions reached. Discuss limitations and the generalizability of
Generalizability, and
Current Knowledge the findings and how the findings fit with current knowledge.
Other
Describe how the study was funded and the role of the funder in the
Source of Funding 23 identification, design, conduct and reporting of the analysis. Describe
other non-monetary sources of support.
Describe any potential for conflict of interest among study contributors
Conflicts of in accordance with journal policy. In the absence of a journal policy, we
24
Interest recommend authors comply with International Committee of Medical
Journal Editors’ recommendations
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
How to Appraise?
Internal Validity Generalizability
• Refers to the strength of • the extent to which the
the inferences from the results of a study apply to
study. individuals and
• E.g. Were costs and circumstances beyond
consequences measured those studied
accurately in • E.g. Was a
appropriate physical comprehensive
units? description of the
competing alternatives
given?
School of
Pharmaceutical
Sciences
How to Appraise: Basic Flow of
& Administrative
Pharmacy
Economic Evaluation
1. Perspective
1. Is it clear?
2. Comparators
1. Is it appropriate?
3. Cost & Consequences
1. Is its measure accurate?
2. Is its value credible?
4. Time Horizon
5. Sensitivity Analysis
1. Is it robust?
6. Discounting
7. Incremental Analysis
School of
Tools for critical

Pharmaceutical
Sciences
& Administrative
Pharmacy
appraisal
• Checklists
– Outline the attributes and standards exhibited
by good quality studies
• Guideline
– Complimentary approach to the use of
checklist
– Attempt to ensure that studies at least
conform to a minimum acceptable standard.
19
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Pharmacoeconomics Literature
Evaluation
• British Medical Journal Checklist
• Checklist to evaluate pharmacoeconomic
studies (Sacristan et al)
• Checklist for assessing economic evaluations
(Drummond et al)
• 6 Principles
• Panel on Cost Effectiveness Checklist (US
public Health Service)
• PhRMA Principles
• And many more... 20
School of
Example: 53 Ec Ev
Pharmaceutical
Sciences
& Administrative
Pharmacy
Asthma Mgmt
Yong YV, Shafie AA. Economic evaluation of enhanced asthma management: a systematic review. Pharmacy Practice. 2014;12(4).
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
• All checklists share a common thread
• Caution: Meeting all criteria in checklists
does not mean the study has validity.
• Critical and informed thinking is vital!
22
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
1. Was a well-defined question posed in an
answerable form?
– Did the study examine both costs and effect of the
service(s)/ programme(s)?
– Did the study involve a comparison of alternatives?
– Was a viewpoint for the analysis stated and was the
study placed in any particular decision-making context?
Example of a well-defined research question:
“ From the perspective of the Ministry of Health, is home-
based palliative care preferable to existing
institutionalized-care in the wards of general hospitals?”
Drummond et al. Methods for the economic evaluation of health care programmes.
3rd Edition. Oxford: Oxford Medical Publications, 2005.
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
2. Was a comprehensive description of the
competing alternatives given?
– Were any relevant alternatives omitted?
– Was (should) a do-nothing alternative (be)
considered?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
3. Was the effectiveness of the programmes or
services established?
– Was this done through a randomized, controlled
clinical trial? If so, did the trial protocol reflect
what would happen in regular practice?
– Were effectiveness data collected and
summarized through a systematic overview of
clinical studies? If so, were the search strategy
and rules for inclusion or exclusion outlined?
– Were observational data or assumptions used to
establish effectiveness? If so, what are the
potential biases in results?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
4. Were all the important and relevant costs
and consequences for each alternative
identified?
– Was the range wide enough for the research
question at hand?
– Did it cover all relevant viewpoints? (Possible
viewpoints include the community or social
viewpoint, and those of patients and third party
payers. Other viewpoints may also be relevant
depending upon the particular analysis.)
– Were capital costs, as well as operating costs,
included?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
5. Were costs and consequences measured accurately
in appropriate physical units (for example, hours of
nursing time, number of physician visits, lost work-
days, gained life-years)?
– Were the sources of resource utilization described and
justified?
– Were any of the identified items ommited from
measurement? If so, does this mean that they carried no
weight in subsequent analysis?
– Were there any special circumstances (for example, joint
use of resources) that made measurement difficult?
Were these circumstances handled appropriately?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
6. Were costs and consequences valued credibly?
– Were the sources of all values clearly identified?
(Possible sources include market values, patient or client
preferences and views, policy makers’ views and health
professionals’ judgement?)
– Were market values employed for changes involving
resources gained or depleted?
– Where market values were absent (for example,
volunteer labour), or market values did not reflect actual
values (such as clinic space donated at a reduced rate),
were adjustments made to approximate market values?
– Was the valuation of consequences appropriate for the
question posed (that is, has the appropriate type of types
of analysis-CE, CU, CB- been selected?)
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
7. Were costs and consequences adjusted
for differential timing?
– Were cost and consequences that occur in
the future ‘discounted’ to their present
values?
– Was any justification given for the discount
rate used?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
8. Was an incremental analysis of costs and
consequences of alternatives performed?
– Were additional (incremental) costs
generated by one alternatives over another
compared to the additional effects, benefits
or utilities generated?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
9. Was allowance made for uncertainty in the
estimates of costs and consequences?
– If patient-level data on costs or consequences
were available, were appropriate statistical
analyses performed?
– If a sensitivity analysis was employed, was
justification provided for the ranges or
distributions of values (for key study parameters),
and the form of sensitivity analysis used?
– Were the conclusions of the study sensitive to the
uncertainty in the results as quantified by the
statistical and/ or sensitivity analysis?
School of
Pharmaceutical
Sciences
Checklists for
& Administrative
Pharmacy
Evaluation
10. Did the presentation and discussion of study results include all
issues of concern to users?
– Were the conclusion of the analysis based on some overall index or
ratio of costs to consequences (for example, cost-effectiveness ratio)? If
so, was the index interpreted intelligently or in a mechanistic fashion?
– Were the results compared with those of others who have investigated
the same question? If so were allowances made for potential
differences in study methodology?
– Did the study discuss the generalizability of the results to other setting
and patient/ client groups?
– Did the study allude to, or take account of, other important factors in the
choice or decision under consideration (for example, distribution of costs
and consequences, or relevant ethical issues)?
– Did study discuss issues of implementation, such as the feasibility of
adopting the ‘preferred’ programme given existing financial or other
constraints and whether any freed resources could be redeployed to
other worthwhile programmes?
School of
Other Additional
Pharmaceutical
Sciences
& Administrative
Pharmacy
Checklist

COSTS

Cost calculations based on reliable databases or data 1
sources conducted for specific study – same
jurisdiction
Recently published cost calculations based on reliable 2
databases or data course – same jurisdiction
Unsourced data from previous economic evaluation – 3
same jurisdiction
Recently published cost calculations based on reliable 4
databases or data sources – different jurisdiction
Unsourced data from previous economic evaluation – 5
different jurisdiction
Expert opinion 6
Not clearly stated 9
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy

UTILITIES

Direct utility assessment for the specific study from a sample either: 1
(a)of the general population
(b)with knowledge of the disease(s) of interest
(c)of patients with the disease(s) of interest
Indirect utility assessment from specific study from patient sample with disease(s) of interest, 1
using a tool validated for the patient population
Indirect utility assessment from a patient sample with disease(s) of interest, using a tool not 2
validated for the patient population
Direct utility assessment from a previous study from a sample either: 3
(a)of the general population
(b)with knowledge of the disease(s) of interest
(c)of patients with the disease(s) of interest
Indirect utility assessment from previous study from patient sample with disease(s) of interest, 3
using a tool validated for the patient population
Unsourced utility data from previous study – method of elicitation unknown 4
Patient preference values obtained from a visual analogue scale 5
Delphi panels, expert opinion 6
Not clearly stated 9

School of
Example: 53 Ec Ev
Pharmaceutical
Sciences
& Administrative
Pharmacy
Asthma Mgmt
Yong YV, Shafie AA. Economic evaluation of enhanced asthma management: a systematic review. Pharmacy Practice. 2014;12(4).
School of
Pharmaceutical
Sciences
& Administrative
Pharmacy
The End

D2L3 - Critical Appraisal Ec Ev v1

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D2L3 - Critical Appraisal Ec Ev v1

Uploaded by

Copyright:

Available Formats

School of

REPORTING & APPRAISAL OF

Assoc Prof Dr Asrul Akmal Shafie

Wagstaff & Culyer (2012) J Health Econs

Drummond (1993) Int J Health Tech Assessment

• European Community Concerted Action on the Harmonization of the

A brief list of recommendations subdivided into the five

3 Provide an explicit statement of the broader context for the study.

Tools for critical

Not clearly stated 9

You might also like