THE VALUE QUESTION
HOW TO REALLY MEASURE
COST-EFFECTIVENESS
Henry Glick, Ph.D.
University of Pennsylvania
32 Annual Symposium
“Clinical Microbiology Update 2001”
IMPLICATIONS
• The need to make comparisons with other
diseases and to know the value of the outcome
implies final outcomes more important than
intermediate ones
– Years of life saved
– Quality-adjusted years of life saved (QALYS)
• Other outcomes (e.g., cost per correct
diagnosis) can be calculated, but will be of less
use when arguing for the value of medical
technologies
DECISION ANALYSIS
• Formal technique for combining data
from several sources using simulation
models
• Abstract relevant parts of reality
• Forces the identification of options and
potential outcomes
• Forces decision maker to make
assumptions explicit
• Is a medical technology good value for the
cost?
– Are buying sufficient health for what we are
spending?
* Comparison of costs and effects with other therapies for
the same disease
* Comparison with other therapies for different diseases
• Need to know the value of the outcome we
are assessing
– Is spending $5,000 to increase the accuracy of a
test by 1 percentage point good value or not?
COMMON METHODS
• Decision analysis
e.g., Frazier et al. Cost-effectiveness of screening for
colorectal cancer in the general population. JAMA
2000;284:1954-61.
• Randomized trials
• References: Economic Evaluation in Health
Care: Merging Theory with Practice. Ed. M
Drummond and A McGuire. Oxford University
Press, 2001.
Cost-Effectiveness in Health and Medicine. Ed. MR
Gold, et al. Oxford University Press, 1996.
TYPES OF MODELS
• Decision trees
– Graphic representation of a set of
equations
• Survival function models
• State transition or Markov models
1
 Treat Vs. Test and Treat STEPS IN THE DEVELOPING A
DECISION TREE
Disease Test + $26,000
+ 0.9 5 QALYS

Test and 0.2 Test - $10,000

1. Develop the structure of the model (e.g.,
Treat 0.1 1 QALYS draw the decision tree)
Disease Test + $16,000
- 0.15 14 QALYS 2. Identify probabilities that different
0.8 Test - $1,000
events in the model occur
Treatment
0.85 15 QALYS
Decision Disease
+ $25,000
3. Identify the outcome values
0.2 5 QALYS
Treat
Empirically
4. Analyze the tree (evaluate the principal
Disease
analysis / analyses)
- $15,000
0.8 14 QALYS 5. Perform sensitivity analysis

Treat Vs. Test and Treat SOURCES OF PROBABILITIES

• Published reports of clinical and
Disease Test + $26,000
+ 0.9 5 QALYS
epidemiological studies
Test and 0.2 Test - $10,000 – Intervention Study
Treat 0.1 1 QALYS * Natural history of disease
Disease Test + $16,000
* Effect of intervention
- 0.15 14 QALYS

0.8
Treatment
Test - $1,000 – Intervention Study and Observational Study
0.85 15 QALYS
Decision Disease * Natural history: observational study
+ $25,000 * Intermediate effect of intervention: intervention study
Treat 0.2 5 QALYS
* Final effect of intervention: observational or intervention study
Empirically
Disease
-
• Consensus of expert opinion
$15,000
0.8 14 QALYS • Personal opinion

IDENTIFYING THE OUTCOMES RESULTS

• Costs
– Direct medical costs • For each strategy:
– Direct nonmedical costs – Expected costs
– Indirect costs – Expected outcomes
• Effects
– Survival • Comparison of costs and effects
– Years of survival – Difference in expected costs
– Quality-adjusted years of survival – Difference in expected effects
– Intermediate measures such as – Ratio of difference in expected costs and
sensitivity/specificity/ likelihood ratios effects

2
 SENSITIVITY ANALYSIS POTENTIAL LIMITATIONS
• When events can repeat themselves
• Confidence intervals (e.g., when one can develop cancer this
– “Second-order” Monte-Carlo simulation year, next year, etc.), decision trees
may become too “bushy”
• Robustness of results • Trees generally don’t directly account
for timing of events (e.g., that one can
• Threshold analysis develop cancer this year, next year,
etc.)
• Markov or state transition models may
help overcome these problems

MARKOV OR STATE Colorectal Cancer States

TRANSITION MODELS No
Polyps
• Describe the disease process by simplifying it into a set Disease
of states

• Disease progression represented probabalistically as a

set of transitions among the states during fixed intervals Local
Death
of time (e.g., 6 month or 1 year intervals) CA

• Effects of an intervention (e.g., a diagnostic test to

detect polyps and cancer) described as a change in the
transition probabilities Regional Distant
CA CA
• Assess resource use and QALYS as a function of being
in a state for a period

STEPS IN DEVELOPING A RANDOMIZED TRIALS

MARKOV MODEL
• Considered the gold standard for evaluating the
1. Enumerate the states efficacy, effectiveness, and efficiency of medical
technologies
2. Define allowable state transitions
• Potential advantages:
3. Associate probabilities with the transitions – Provide head-to-head comparison between technology
under evaluation and alternative
4. Identify a cycle length
– Collect costs and outcomes in same individuals, and
5. Identify an initial distribution of patients within the states maintain correlations between costs and outcomes

6. Identify the outcome values • Potential disadvantages

– May reflect idealized rather than usual practice
7. Analyze the Markov model
– May not provide sufficient follow-up to determine final
8. Perform sensitivity analysis outcomes

3
 STEPS IN CONDUCTING ECONOMIC
EVALUATION IN RANDOMIZED TRIAL
• Step 1: Quantify the costs of care
• Step 2: Quantify outcomes
• Step 3: Assess whether and by how much average costs
and outcomes differ among the treatment groups
• Step 4: Compare magnitude of differences in costs and
outcomes and evaluate "value for the cost"
(e.g., by reporting a cost effectiveness ratio or the
probability that the ratio is acceptable)
• Step 5: Conduct sensitivity analysis
FOUR ISSUES
• What health care resource use should be
measured (data collection)?
• How should the resources be valued?
• How many subjects should be included in the
study (sample size)?
• For time-limited trials of therapies with longer-
term effects: How should long-term results be
projected?

WHAT HEALTH CARE RESOURCE USE HOW SHOULD THE

SHOULD BE MEASURED?
• Ideal: All resource use
• In practice:
– Services that make up a large portion of the
difference in treatment between patients randomized
to the different therapies under study
* Estimate of the cost impact of the therapy
RESOURCES BE VALUED?
• SOURCES OF U.S. UNIT COST DATA
– Hospital charges adjusted using cost to
charge ratios
– Internal hospital costing system data
– Diagnosis related group payments (Hospital
stays)

– Services that make up a large portion of the total
* Reduces likelihood that differences among unmeasured
services will lead to bias
* Provides measure of variability of costs
– Resource-based relative value units
(Physician services/procedures)
– Trial-specific costing exercise (e.g., time and
motion studies

HOW MANY SUBJECTS SHOULD BE SAMPLE SIZE DATA

INCLUDED IN THE STUDY?
• Early 1990's approach: Select the larger of the
sample sizes needed for estimating pre-specified
cost and effect differences
– i.e., what sample size was required to identify a $1000
difference in costs, and what was required to identify a
10% reduction in mortality
• Current approach: Estimate the number of study
subjects needed to rule out unacceptably high
upper confidence limits for the cost-effectiveness
ratio
REQUIREMENTS
• Difference in costs (old and new)
• Difference in effects (old and new)
• Standard deviation of costs (old and
new)
• Standard deviation of effects (old and
new)
• Correlation of costs and effects (new)

4
 15000
$6,277 HOW SHOULD LONG-TERM
$22,971 RESULTS BE PROJECTED?
12000
• For time-limited trials of therapies with potentially
Incremental Costs

9000 $4,407 long term effects, one should evaluate the costs
and outcomes that were observed during the trial
6000
– Maintain the same time horizons for costs and
$1,140
3000 outcomes observed in the trial

0 • Because the relative magnitude of incremental

costs and outcomes observed during the trial may
-3000
-0.10 0.40 0.90 1.40 1.90 2.40
not be reflective of the relative magnitude for
Incremental QALYS
longer periods of follow-up, one should also
Point Estimate CER: $5700
Means and S.D.s for identical project the results for longer periods
Correlations: 0.988; -0.988

COST EFFECTIVENESS RATIO AND 95% CONCLUSIONS

CI, 4S FOR DIFFERENT LENGTHS OF
FOLLOW-UP/PROJECTION
• Economic evaluations attempting to assess
_____________________________________________________
value for the cost
Years of
Follow-up Point Estimate 95% CI • Given need to compare interventions for a
_____________________________________________________
single disease and among diseases, a
2 $282,857 $45,577 to Dominated common outcome such as survival or quality
adjusted survival should be used
4 $12,074 Dominates to Dominated
• Common techniques for evaluation include
5.5 $15,258 Dominates to $122,772
decision analysis and randomized trials
10 $12,246 Dominates to $42,263 • Lots of opportunities exist for evaluating
20 $7,320 $681 to $21,841 microbiologic technologies
_____________________________________________________