Real-World Evidence for

Decision-Making

Mark McClellan, MD, PhD

Robert J. Margolis Professor of Business, Medicine, and Policy
Director, Duke-Margolis Center for Health Policy
Dr. McClellan is an independent member of the Board of Directors for
1
Alignment Healthcare, Cigna, Johnson & Johnson, and PrognomIQ
We Need Better Evidence to Improve Health Care And
Reduce Its Costs

Enhanced Prevention – Earlier and More Accurate

Behavioral and Social Influences Diagnosis
on Health and Well-Being

More Efficient and Effective

More Definitive Treatments for Supportive Care for Remaining
Disease Interception or Cure Health Gaps and Functional
Impairments

2
The Potential of Real-World Data and Real-World Evidence
RWD/RWE can improve the efficiency
of clinical research
• Traditional clinical trials are increasingly
time- and resource-intensive to conduct
• Growing base of RWD from electronic health
information infrastructure has enabled
routine and increasingly robust collection of
digital data at the point of patient care
RWD/RWE can be used to determine
effectiveness of medical products and
practices
• RWE can derive from more diverse patient
populations and care settings
• Clinical trials do not generate evidence on all
treatments and outcomes that matter for
patients, providers, and payers
• RWE can help fill remaining downstream evidence
gaps
3
 Vast and
Growing
Health Data
That Could
Inform
Care

Compiled by Datavant: https://medium.com/datavant/the-fragmentation-of-health-data-8fa708109e13 4

Duke-Margolis Study of Regulatory Use of RWE

                            
BRIEF REPORT

Understanding Use of Real‐World Data (RWD) and Real‐World Evidence

(RWE) to Support Regulatory Decisions on Medical Product Effectiveness
Nirosha Mahendraratnam 
 Kerra Mercon 
 Mira Gill 
 Laura Benzing 
 Mark B. McClellan
First published: 23 April 2021
 

https://doi.org/10.1002/cpt.2272

5
Main findings on regulatory use
• Review of evidence packages for approvals 1954- June 2020 identified 34 uses
• Excluded cases that only used RWE for safety reporting
• 26% oncology, 18% hematology, 12% neurology
• 50% rare condition, 82% received orphan designation
• 61% included RWE on label
• Majority of uses were for external controls in single-arm trials
• Stated reasons not included: lack of prespecification of study design, data issues raising
questions of reliability, relevance of RWE analyses
• Broader industry use and acceptance of RWD and RWE will require attention to
whether data and methods are fit for purpose, and more intentional efforts to
develop and learn from promising examples and uses
6
Overview
• Real World Data and Real World Evidence: What and Why
• Advancing Real-World Data, Methods, and Infrastructure
• Impact of COVID-19 and Digital Health
• Next Steps

7
Key Terms
• Real world data (RWD) is data
relating to patient health status
and/or the delivery of health care
routinely collected from a variety
of sources
• Electronic health records
• Payer claims data
• Registries
• Mobile apps and digital technologies
• Patient-generated data
• Real-world evidence (RWE) is
evidence derived from RWD through
the application of research methods
• This includes clinical evidence
regarding the use and potential
benefits or risks of a medical
product derived from analysis of
RWD used for FDA regulatory
decisions, as well as a range of other
uses
8
RWE and data science are expanding
Lifecycle Approach for Real-World Data and Evidence

1. Medical Product 3. Care Delivery 4. Value-Based

2. Regulatory Review
Development Payment and Coverage
• AI-enabled CDS to
• Inform biological • Inform new approvals personalize dx and • Increase stakeholder
understanding of in rare diseases tx decisions understanding of
disease through value of technology
• Inform indication and • Support patients’ by incorporating RWE
registries
labeling decisions engagement in their
• Identify unmet need own care decisions • “De-risk” payment for
• Inform PM safety high cost treatments
• Improve RCT • Help drive higher- to increase access
recruitment efficiency value care

9
RWE has value for Industry:
multiple stakeholders Confirmatory evidence
Continued innovation
Detailed safety/efficacy
profiles
Payers: Regulators:
Informed coverage and Postmarket safety data
reimbursement decisions Informed B-R profiles
Support for increased value Richer subgroup information
Quality improvement Confirmation of surrogate
PRECISION endpoint validity

EFFICIENCY
VALUE

Providers: Patients:
Improved treatment decisions Participation in care and
Quality improvement research
Population health Improved treatment
management decisions

10
Recent legislation directs FDA to explore RWE
Prescription Drug User Fee Act VI
• Required FDA to enhance use of RWE for use in
regulatory decision-making
• FDA must:
o Hold a public workshop with key stakeholders
(e.g., patients, industry, academia) by the end of
2018
o Initiate (or fund) activities (e.g., pilot studies or
methodology development projects) aimed at
addressing key concerns and considerations in
the use of RWE by the end of 2019
o Issue draft guidance by the end of 2021
11
21st Century Cures Act
• Required FDA to establish a program to
evaluate the potential use of RWE to:
o Help support the approval of new
indications for an approved drug
o Help support or satisfy post approval study
requirements
• FDA must issue:
o A draft framework for this program by the
end of 2018 (complete)
o Draft guidance by the end of 2021
11
 COVID-19 Experience with Limitations of Randomized Controlled
Trials: Big Gaps and High Costs for Developing Actionable RCT
Evidence
● Only 5% of COVID-19 trial arms yielded
actionable evidence

● Existing trial infrastructure not equipped to flex

with surges in COVID-19 cases

● COVID-19 therapeutic trials enrolled <1% of

hospitalized patients

● Only 26% of patients who were enrolled in

COVID-19 trials were enrolled in randomized,
adequately powered trials

Reference: https://www.nature.com/articles/d41573-021-00037-3 12
RWE Cautions: Surgisphere paper retractions
• These papers purportedly drew data from a multinational
registry to evaluate the real-world effect of potential
therapies for COVID-19
• Findings from these papers quickly influenced
international COVID-19 response efforts
• But… it soon became evident that the reliability of the
database use in the papers was suspect, and independent
reviewers were not granted access to verify the database
• Identifying consistent mechanisms to assess the reliability
of RWD and determine data fitness-for-use is critical to
building trust

13
Many Collaborative Efforts on RWD/RWE since 2015
• Given legislative mandates in 21st Century Cures for FDA guidance on use of RWE,
FDA and collaborators have focused on policy and technical components
• Many organizations – including Duke-Margolis, Friends of Cancer Research, Clinical
Trials Transformation Initiative, and others – have conducted analyses and
developed recommendations on a range of RWD/RWE issues:
• Assessing RWD quality and reliability
• Characterizing replicability of randomized controlled trial findings in nonrandomized RWE
settings
• Building RWD-supported endpoints, control arms, etc.
• Assessing RWE as part of an evidence package

14
 Duke-Margolis RWE Collaborative Advisory Group*
Data Curators Sponsors Expert Collaborators

*US Food and Drug Administration (FDA) and National Academies

of Sciences, Engineering, and Medicine act as observers

*This project is made possible through the generosity of the Margolis Family Foundation, which provides core resources for the Center, as well as a combination
of financial and in-kind contributions from Collaborative members including AbbVie; Amgen; Boehringer Ingelheim; Eli Lilly and Company; Genentech, a member 15
of the Roche Group; GlaxoSmithKline; Merck; Novartis; Pfizer; and Teva.
RWE Collaborative White Papers

16
 Generating RWE Fit for Regulatory Purpose

Matching data sources and methods to answer specific clinical and regulatory
questions will dictate applicability of RWE for different regulatory use cases
Characterizing RWD Quality and Relevancy for Regulatory Purposes (2018).
17
https://healthpolicy.duke.edu/sites/default/files/atoms/files/characterizing_rwd.pdf
FDA Released the RWE Framework in December 2018
(PDUFA VI and Cures Requirement)
1. Are the RWD are fit for use?
2. Is the trial or study design used to generate RWE
can provide adequate scientific evidence to answer
or help answer the regulatory question?
3. Does the study conduct meets FDA regulatory
requirements for good research practices (e.g., for
study monitoring and data collection)?

18
 FDA’s Technology Modernization Action Plan
(TMAP)
• Released in 2019, the TMAP outlines a plan to:
• Modernize FDA’s technical infrastructure
• Enhance FDA’s capabilities to develop
technology products that support its
regulatory mission; and
• Communicate and collaborate with
stakeholders to drive technological progress
that is interoperable across the system and
delivers value to consumers and patients

https://www.fda.gov/about-fda/reports/fdas-technology-modernization-action-plan 19
 FDA’s Data Modernization Action Plan
(DMAP)
• The DMAP builds on the TMAP and proposes a
framework to:
• Identify and execute high value driver
projects for individual centers and for the
Agency;
• Develop consistent and repeatable data
practices across the Agency; and,
• Create and sustain a strong talent network
combining internal strengths with key
external partnerships

https://www.fda.gov/about-fda/reports/data-modernization-action-plan
20
FDA’s 2021 RWE Agenda
• The FDA’s Center for Drug Evaluation and Research has announced four guidance documents
anticipated this year
• Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory
Decision-Making for Drug and Biological Products
• Regulatory Considerations for the Use of Real-World Data and Real-World Evidence to Support Regulatory
Decision-Making for Drugs and Biological Products
• Using Registries as a Real-World Data Source for FDA Submissions
• Meeting the Substantial Evidence Standard Based on One Adequate and Well-Controlled Clinical
Investigation and Confirmatory Evidence
• User fee negotiations continue toward late summer commitment letter
• Strong agency and stakeholder interest in clinical studies (including randomized trials)
conducted reliably in routine clinical practice settings
• Continued progress on Tech and Data Modernization Plans

21
 Advancing
Real-World Data, Methods, and
Infrastructure

22
• Explores the feasibility challenges to
conducting interventional studies in the real
world including ethical, operational, and
resource barriers
• Highlights key considerations for how to
demonstrate the credibility of an
observational study using RWD

23
 How do demonstrate the credibility of an
observational study for generating real-world
evidence?
Adequate and Well-Controlled Substantial Evidence
• What are the characteristics of a study that • What are the relevant factors for determining
can be considered adequate and well- substantial evidence?
controlled?
• Treatment effect size?
• Appropriate comparisons?
• Multiple studies with consistency?
• Balanced groups?
• Strength of relevant prior evidence?
• Adequate control for observed biases? What
• Regulatory flexibility due to the disease or high
about unobserved?
unmet need?

Understanding the Need for Non-Interventional Studies Using Secondary Data to Generate Real-World Evidence for Regulatory Decision Making,
and Demonstrating Their Credibility (2019). 24
https://healthpolicy.duke.edu/sites/default/files/2020-08/Non-Interventional%20Study%20Credibility.pdf
COVID-19 Disrupted the Traditional Evidence
Generation Paradigm

RWE from Digital Tools Real-World

Practical and Data
Trials Technology Infrastructure
Acceleration of
Large-scale adoption of digital Stakeholder
practical trials for tools, remote collaboration to
COVID-19 patient monitoring, align observational
therapeutics and telemedicine studies for COVID-
in COVID-19 19 therapeutics

25
 Target trials
• Target trial: a hypothetical randomized trial that could be conducted to answer a question

• By emulating a target trial with observational data, we can ensure that our research question is
well defined
• As part of the process, a target trial protocol is specified, and then emulated with observational data.

• Explicitly emulating a target trial helps control for common biases.

• The target trial approach works best for scenarios that are not-placebo controlled or blinded as
observational data is poorly suited to emulating trials designs that incorporate these approaches.

See more in: Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not
Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254 26
Example of emulation of target trial:
Tocilizumab in critically ill COVID-19 patients

• Gupta et al. JAMA Internal Medicine October 2020

• Much off-label use early in pandemic in patients with evidence of
hyperinflammation, based on early observation of increased risk of
death in patients with elevated IL-6 levels (Tocilizumab is mAb
against IL-6 receptor)
27
Example of emulation of target trial:
Tocilizumab in critically ill COVID-19 patients
• Gupta et al. JAMA Internal Medicine October 2020
• Followed key principles of emulating a target trial:
• Defined target trial: use of Tocilizumab in 3924 COVID-19 patients admitted to ICU
• Sampled cases and controls from this cohort in well-defined RWD platform with rich covariates
• Adjusted for confounding (part of emulating randomization)
• Specified time zero: time when eligibility criteria met, treatment strategy is assigned, and study
outcomes begin to be counted (Hernán et al. J Clin Epidemiol 2016; 79:70-75)
• Addresses selection bias and immortal time bias

• Found 30-day mortality hazard ratio of 0.71 (CI 0.56-0.92), with 0.41 if admitted to ICU
within 3 days of symptom onset and 0.85 if after 3 days
• Subsequent randomized clinical trials (RECOVERY, REMAP-CAP) confirmed these findings

28
Why Don’t We Have More Real-World Randomized Trials?
1. Nonsustainable financing — reimbursement is not aligned with data collection
requirements, since business models are designed to accrue returns from care provision
rather than research participation
2. Legal and Regulatory Oversight Issues – liability concerns and indemnification issues,
apprehension to defer to cIRBs, and other regulatory requirements complicate site
contracting; significant clinician time and effort required for research training and trial
compliance requirements; challenges in consenting patients and implementing placebo
controls complicate incorporation in routine practice
3. Complex Data Collection Requirements—complex trial protocols and limited site-based staff
with extensive trial experience make it difficult to adhere to data collection requirements
that extend beyond routine care provision
4. Limited Ability to Leverage Automated Data Capture— capacity for reliable, research-grade
automated capture from patient health records, lab data, patient-generated data, and other
electronic sources means additional burdens for completing electronic case record forms
29
Starting Points for Community-Based Trial Networks
Consider Product Characteristics and Therapeutic Context

Digital
Technology
New Capacity Building Supplemental
Indications
Evidence on
for Existing New Drugs
Regulatory and
Drugs Payment Policy
Reforms

Potential Applications
1. Registries – many health systems and delivery networks are enhancing their capacity to track matched patients longitudinally –
observational RWE research can provide foundation for randomized studies
2. Pandemic Preparedness—platform trials to assess therapeutics for respiratory symptoms of viral infection (building on
RECOVERY Trial approach)
3. Chronic Disease— e.g., comparative effectiveness studies for cardiovascular and neurological diseases
4. Acute Conditions— serious infections with risk of antimicrobial resistance, cardiac events

30
RWE, AI, and Optimizing Care Delivery
• Extensive investment and applications aiming to improve outcomes and
reduce costs of care, using increasingly rich data sources (EMRs, claims,
patient-generated health data, digital and telehealth data, matched non-
medical data):
• Predicting risk of poor outcomes: unplanned admissions, falls, complications after
surgery, etc.
• Predicting response to interventions and combinations of interventions:
• Clinical decision support for diagnosis and treatment selection
• Targeting outreach and interventions to address risks and improve patient experience
• Addressing health disparities and provider biases
• Nature Medicine (Jan. 2021): AI algorithm was better able to predict pain then radiologists in
multiple types of underserved populations (racial, lower income, and less educated patients)
31
 Potential for Bias in AI to Influence Care
• Representativeness
• For AI algorithms to be generalizable, training data needs to be representative of the people and
places where it will be used - demographically, geographically, and by type of provider system
• Many training data sets come from academic hospitals in only 3 states (CA, MA, and NY)
• Biased data
• Consider potential for bias in proxy measures (e.g., cost or utilization proxy for severity of illness)
• Potential for bias in provider-reported data
• Inequities induced by responses to prediction models
• Reacting to a no-show prediction by double-booking the appointment time slot
 Need to test for bias
• Clearly document the patient populations the algorithm was trained and tested on
• Test how well an algorithm predicts its actual target if using a data proxy during training

“The Algorithmic Playbook” at

https://www.chicagobooth.edu/research/center-for-applied-artificial-intelligence/research/algorithmi 32
c-bias/playbook
 Some Barriers to Higher Value in Drug Payment
• Physician-administered drugs (Medicare Part B): Physicians who administer drugs are
reimbursed based on the drug’s average sales price, with no formulary or negotiation to
encourage prices that reflect value or shifts to less costly but effective drugs, and with
higher physician and hospital payments for using more and more costly drugs.
• Oral/self-administered drugs (Medicare Part D): Current benefit structure has no out-of-
pocket maximum, and limited incentives for payers and manufacturers to negotiate lower
prices for specialty drugs – because Medicare pays most of the drug costs in the
“catastrophic” phase of the Part D benefit.
• Fee-for-service prices don’t fit needed innovation: More learning and improvements with
big data and real-world evidence after products on market, effects differ across patients,
new comprehensive care models with payments based on outcomes and lower OOP costs

33
Framework for Increasing Value of Medical Technologies
Volume-Based Pricing and Payment Alternative Payment Models

Category 1: Category 2: Category 3: Category 4:

Strictly Fee- Fee-For- Outcomes- Accountable
For Service Service Tied to Based Payment Population
Value Payment

Examples: Examples: price Example: larger Example:

volume-based linked to cost- or full rebate if “Subscription
discounts, effectiveness a product does payment” (PMPM) to
analysis (ICER), not meet a manufacturer or
generic drug
coverage linked to performance specialty care
competition producing more measures (e.g., manager, may be
evidence on outcomes, linked to population
benefits, risks, and complications). outcomes.
costs (CED).

Paying For Value From Costly Medical Technologies: A Framework For Applying Value-Based Payment Reforms
Marianne Hamilton Lopez, Gregory W. Daniel, Nicholas C. Fiore, Aparna Higgins, and Mark B. McClellan 34
Health Affairs 2020 39:6, 1018-1025
Payer support for evidence development
• Medicare’s Coverage with Evidence Development (CED) has been used to support post
market registries, to augment evidence on drug utilization in Medicare populations,
expand label indications, and identify patient factors for success for novel therapies
• Recent proposal for Medicare Coverage for Innovative Technologies (MCIT), now under
revision by CMS, reflects strong interest in supporting access and further evidence
development on breakthrough technologies
• Private payers have implemented reporting requirements related to outcomes and
performance for Center of Excellence determinations that affect coverage and
payment, and have sought to restrict coverage for new technologies pending additional
evidence development

35
RWE and data science are expanding
Lifecycle Approach for Real-World Data and Evidence

1. Medical Product 3. Care Delivery 4. Value-Based

2. Regulatory Review
Development Payment and Coverage
• AI-enabled CDS to
• Inform biological • Inform new approvals personalize dx and • Increase stakeholder
understanding of in rare diseases tx decisions understanding of
disease through value of technology
• Inform indication and • Support patients’ by incorporating RWE
registries
labeling decisions engagement in their
• Identify unmet need own care decisions • “De-risk” payment for
• Inform PM safety high cost treatments
• Improve RCT • Help drive higher- to increase access
recruitment efficiency value care

36
Building a Better RWD Ecosystem

Improving data • Facilitate efficient data collection for trials

• Enable implementation of novel trial designs
capture at the
• Feasibly fill evidence gaps for products with
point of care and limited evidence-base
aligning data • Fulfill regulatory post-marketing requirements,
curation to: and payer and patient evidentiary needs

37
Iterative Steps to Apply and Improve RWE

Harmonize Develop Use results and

Identify common common data standardized iterate on
data elements element study design and improving data
definitions analysis plan and methods

38
Opportunities for more efficient and extensive infrastructure to
address FDA and CMS post-market needs
• Further development of digital technology, electronic medical record standards, and
data curation have the potential to improve real-world data collection, reducing
provider burden and enhancing foundation for post market evidence
• Registries maintained by specialty societies (e.g. TVT Registry) and supported by
Coverage with Evidence Development have been used to satisfy FDA postmarket and
CMS coverage expectations, and improve evidence for clinical practice
• Collaborating with health technology firms and other partners, in conjunction with
MCIT implementation, could enhance these capabilities

39
Looking Ahead…

Develop shareable,
Enhance a shared RWD
practical “use cases” Build on learnings from
infrastructure to align
related to effectiveness response to the COVID-
and improve data
to facilitate learning and 19 pandemic and beyond
collection efforts
effective adoption

40
Thank You!
Contact Us Follow Us

healthpolicy.duke.edu DukeMargolis

Subscribe to our monthly newsletter at

dukemargolis@duke.edu @DukeMargolis

1201 Pennsylvania Avenue, NW, Suite 500 @DukeMargolis

Washington, DC 20004

DC office: 202-621-2800
Duke Margolis
Durham office: 919-419-2504

41