SUPPLEMENT ARTICLE
Real-World Evidence: A Primer
Robert Zura, MD,a Debra E. Irwin, PhD, MSPH,b Christina D. Mack, PhD, MSPH,c
Molly L. Aldridge, MPH,d and John I. Mackowiak, PhDe
Downloaded from http://journals.lww.com/jorthotrauma by A3VIojwJQicSF5ZP0PJN0CQE9JYDLq1/0+M8cGuiE1SZpIj9Sk6XhhL+QSWshocKIiaXqlKc4I4vBiZLZMWS0UcNBGYxhomytUsWiZExWDMWq+BYq88V4G7TNPrjn5V+kVuVYeLQE5Q= on 03/26/2021
Summary: The insights that real-world data (RWD) can provide,
beyond what can be learned within the traditional clinical trial
setting, have gained enormous traction in recent years. RWD, which
are increasingly available and accessible, can further our under-
standing of disease, disease progression, and safety and effectiveness
of treatments with the speed and accuracy required by the health care
environment and patients today. Over the decades since RWD were
first recognized, innovation has evolved to take real-world research
beyond finding ways to identify, store, and analyze large volumes of
data. The research community has developed strong methods to
address challenges of using RWD and as a result has increased the
acceptance of RWD in research, practice, and policy. Historic
concerns about RWD relate to data quality, privacy, and trans-
parency; however, new tools, methods, and approaches mitigate
these challenges and expand the utility of RWD to new applications.
Specific guidelines for RWD use have been developed and published
by numerous groups, including regulatory authorities. These and
other efforts have shown that the more RWD are used and
understood and the more the tools for handling it are refined, the
more useful it will be.
Key Words: real-world data, real-world evidence, big data
(J Orthop Trauma 2021;35:S1–S5)
INTRODUCTION
The story of real-world data (RWD) is decades in the
making. Scholars acknowledged the increasing rate of growth
of the volume of data by coining the term “information explo-
sion” as early as 1941. “Big data” is a term used to describe
data sets that are too big or too complex to be handled by
Accepted for publication December 9, 2020.
From the aDepartment of Orthopaedics, Louisiana State University Health
Sciences Center, New Orleans, LA; bIBM Watson Health, Durham, NC;
c
IQVIA Real-World Solutions, Durham, NC; dAldridge Medical Writing &
Consulting, Chapel Hill, NC; and eCenter for Outcomes Research, Cedar
Point, NC.
R. Zura is a consultant for Bioventus, Stryker, and Osteocentric. He is a board
member for the Kuntscher Society and the Louisiana Orthopaedic
Association and serves on the Editorial Board of Orthopedics. D. E.
Irwin is an employee of IBM Watson Health, and IBM Watson Health
received financial support from Bioventus as a consultant on the BONES
Study. C. D. Mack is a full-time employee of IQVIA which is a research
organization contracted with Bioventus on the BONES study. She holds
stock in Johnson & Johnson and AstraZeneca. M. L. Aldridge is an
independent medical writing consultant contracted with Bioventus for this
publication. J. Mackowiak is a consultant contracted with Bioventus for
this publication.
Reprints: John I. Mackowiak, PhD, Center for Outcomes Research, 186 Bluff
Road, Cedar Point, NC 28584 (e-mail: jmackowiak@c-o-r.org).
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
DOI: 10.1097/BOT.0000000000002037
J Orthop Trauma  Volume 35, Number 3 Supplement, March 2021
Copyright © 2021 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
traditional processing machines and applications.1 In medicine,
big data often means RWD and includes information on “patient
health status and/or the delivery of health care routinely collected
from a variety of sources.2 These “vast and growing volumes” of
digital health information come in many forms including elec-
tronic health records (EHRs), administrative or health claims
data, disease and drug monitoring registries, information from
wearable technology, and many others.3–6
Another term, “real-world evidence (RWE),” describes
“clinical evidence about the usage and potential benefits or risks
of a medical product derived from analysis of RWD.”2 A 2018
benchmark study conducted by the Tufts Center for the Study of
Drug Development found that of 30 pharmaceutical companies
surveyed, 29 had an RWE function, most of which were increas-
ing in size. Developing greater insight into therapeutic area
needs and a better understanding of drug effectiveness were 2
of the cited measures of return on their investment.7
The promise of RWE is one of great opportunity for
expanded approaches, more complete views, and new angles
of understanding of human health and medicine. This article
reviews some of the current opportunities and challenges
associated with the use of RWD and RWE in medicine.
Study Designs For Use With RWE
RWD have been used across many simple and complex
study designs. Case–control studies identify groups of people
with and without an outcome of interest and compare them
regarding their exposure status. Cohort studies identify sub-
sets of a population with and without an exposure the
investigators believe may influence the probability of a par-
ticular outcome. A case–control study of 1021 patients with
hip fractures (cases) and 5606 controls among elderly
Medicaid enrollees identified an increased risk of falling and
fractures (outcomes) in elderly persons using psychotropic
drugs (exposure).8 Lin et al9 used data from national health
research datasets to conduct a cohort study estimating inci-
dence and factors associated with hip fractures in Taiwan.
Epidemiologic studies often use a cross-sectional design with
RWD to estimate prevalence. Cross-sectional studies examine
the relationship between an outcome and exposures as they
exist in a defined population at a particular time.
Pragmatic clinical trials are gaining interest across
several therapeutic areas. These studies can incorporate
randomization, compare a treatment of interest with 1 or
more treatments being used in the real-world setting, and
examine clinical events that occur through regular, real-world
medical care.6
As an alternative to randomized experimental designs,
studies using propensity score matching make use of
www.jorthotrauma.com | S1
Zura et al J Orthop Trauma  Volume 35, Number 3 Supplement, March 2021
quasiexperimental comparison groups. A variety of propen-
sity score techniques can be used to identify nonconcurrent
comparison subjects that are similar to the treated subjects
in a separate study and can also be used to further adjust
observable differences.10 As an example, propensity score–
matched cohorts were created using a nationwide inpatient
database to evaluate in-hospital complications after microen-
doscopic versus open lumbar laminectomy in Japan.11
Analytical Methods and Tools
The use of RWE in these study designs is supported by
innovative methods, including tools such as predictive
modeling, machine learning, and text mining, which can turn
“big data” into “smart data.”12
Developments of new and improved statistical methods
are coming with nearly the volume, velocity, and variety of
the data that necessitated their creation. For example, pre-
dictive modeling can estimate the likelihood of occurrences
and can help identify groups of patients who are more likely
to respond to an intervention.1 These techniques are not lim-
ited to the same requirements as more traditional statistical
methods and allow for the analysis of not only very large
numbers of observations, but also very large numbers of var-
iables. Predictive analytics have been used widely for pur-
poses ranging from the prediction of cartilage lesion
progression in knee osteoarthritis13 to the prediction of mor-
tality in the intensive care unit.14 RWD can be used to eval-
uate the predictive value of risk assessment tools, as was
demonstrated by the use of an administrative claims database
to evaluate the World Health Organization’s (WHO) Fracture
Risk Assessment Tool (FRAX).15
Machine learning is a discipline that focuses on how
computers learn from data and the development of algorithms
that make this learning possible.1,12,16 Used numerous times
in orthopaedics,17 applications of RWE and machine learning
are documented as far back as 2005 and range from automatic
interpretation of medical images (an example of “supervised
learning”) to the practice of precision medicine which
addresses the enormous, inherent heterogeneity of many com-
mon diseases (an example of “unsupervised learning”).4,16
Text mining techniques such as natural language processing
allow for the analysis of data from clinical notes to corrobo-
rate more structured data in medical records which may be
prone to errors.12,18
It should be recognized that some tools for the analysis
of big data sets can be easily accessed and used by
nonexperts. This can lead to incorrect or unreliable interpre-
tation or use of evidence. To this end, design and conduct of
RWE studies and subsequent publication of results should be
conducted by those with methodological expertise.5
How Can RWD Be Used?
RWE is poised to profoundly improve our understand-
ing of human health. One of the most obvious contributions
of RWE to medicine is its size and scope, enabling a unique
set of opportunities previously impossible to pursue with
smaller or more circumscribed data sets. From biomarker
discovery and validation to trial feasibility to safety signal
detection, the uses of RWD are ever increasing. Below a
S2 | www.jorthotrauma.com
Copyright © 2021 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
number of RWE uses that are of particular interest in
orthopaedics are discussed.
Safety Monitoring/Pharmacovigilance
RWE from observational studies has long been
accepted for postapproval safety monitoring and to answer
pharmacoeconomic questions.19 The role of RWD in pharma-
covigilance includes drug safety signal detection, evaluation,
substantiation, and validation.4,20 Pharmaceutical and device
surveillance in the United States relies heavily on spontane-
ous reporting systems to document adverse events (AEs) and
mandatory reporting from manufacturers and physicians to
the US Food and Drug Administration (FDA), but other sour-
ces of RWD are also common.18 Dasa et al21 used a large
claims database to monitor AEs and time to total knee
replacement after repeated courses of hyaluronic acid injec-
tions. FDA uses various sources of health care information in
its Sentinel database which was designed to support postap-
proval safety efforts.22 Similarly, the National Evaluation
System for health Technology (NEST) uses RWD for surveil-
lance of devices in clinical care.23
Quality Monitoring
Many RWD sources, including insurance claims and
EHR databases, are used to monitor quality of care.24 The
National Quality Forum offers several quality metrics, includ-
ing both process-related and outcome-related metrics for
orthopaedic care,25 many of which can be monitored using
RWD sources. The use of appropriate outcomes and statistical
methods that account for the complex and dynamic range of
factors that influence patient outcomes and the acceptance of
public reporting as an opportunity for quality improvement
may increase trust and appreciation of performance monitor-
ing data among current critics.26,27
External Comparators
External comparator groups, sometimes called historic
or synthetic controls, are patient cohorts created from data
that are external to the clinical study.5,28 These data can be
used to augment clinical trials, in particular single-arm trials
when a comparison group is not feasible. Data on external
comparators can be collected either before or concurrent with
the main study and can come from a variety of RWD sour-
ces.28 For example, the Bioventus Observational Non-
interventional EXOGEN Studies (BONES) program used
data from a large administrative claims database to create a
comparison group for the patients prospectively enrolled in its
registry study of a fracture healing device.28 A recent system-
atic review identified 43 occasions in which nonrandomized
study designs using external controls were included in appli-
cations to the European Medicines Agency (EMA) and/or
FDA, most of which were met with approval.28
Reaching a Broader Population
One of the key advantages of RWD is that it provides a
diverse, generalizable population and access to patients who
are not in clinical trials. To reduce confounding and maximize
validity, clinical trials often include very narrow populations
in which to test the efficacy and safety of interventions.6
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
J Orthop Trauma  Volume 35, Number 3 Supplement, March 2021
Studies using RWE enable inclusion of broader patient
groups without the constraints of trial inclusion and exclusion
criteria.19,28,29 Reaching a broader population allows for cap-
ture of rare events, such as nonunions, infections, and pulmo-
nary embolism after arthroplasty.
Improve Efficiency of Clinical Trials
RWD from EHRs may allow clinical investigators to
access and combine data from clinical notes and laboratory
and pharmacy records to improve efficiency of data collection
and analysis, including post-trial follow-up.5 Exploratory
analyses of subgroup effects are generally not encouraged
unless defined at study outset, and sufficient sample size is
required to detect any subgroup effects.6 RWD can facilitate
creation of subgroups for exploratory analyses. Efficiency is
also achieved through use of external comparators, particu-
larly in rollover trials, cases where randomization may not be
feasible or ethical, or where there are limited number of
patients with the condition of interest.19
Regulatory Approvals
The use of RWD as historical controls has supported
successful regulatory applications for nearly 50 years,19 but
regulatory decision-making has relied primarily on data from
RCTs which can be expensive and time-consuming.
Importantly, RCTs often have limited applicability to real-
world settings, reflecting smaller populations with limited
follow-up and a lack of generalizability. FDA affirms their
commitment to improvement of the drug approval process
through publication of the framework for their RWE program,
release of guidance on the use of RWD to support regulatory
decision-making for medical devices, and through the launch
of numerous initiatives to support the application of RWE in
regulatory decision-making.2,19,23 Hybrid trial methodologies
including pragmatic RCTs and single-arm trial data supple-
mented with RWD external comparators combine the benefits
of RCTs (including the potential for randomization) with
those of observational study designs and are, with increasing
frequency, being used to create RWE appropriate for regula-
tory decision-making.6,19 The recent label expansion for
Ibrance (palbociclib, Pfizer) was based solely on RWE, but
RWE has complemented clinical trial data to drive label
expansions for several other drugs19,30,31 and can further be
used for postapproval surveillance as a condition of
approval.23 Interestingly, FDA acknowledges that, for regu-
latory purposes, the required level of RWE data quality varies
depending on the specific regulatory decision being made.23
Hypothesis Generation and Predicate Free Analysis
Examination of RWD allows for the detection and
quantification of correlations and subsequent hypothesis
generation.32 Bate et al20 recently discussed the use of
RWD for searching for potential signals across multiple
drug/AE pairs without the need of prior suspicion.
Policy and Reimbursement Decisions
RWD enable policymakers to evaluate the effects of
policy changes and to evaluate future policy decisions.33
Pragmatic clinical trials inform a clinical or policy decision
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2021 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
A Primer on Real-World Evidence
by providing evidence for adoption of the intervention into
real-world clinical practice.5,19,30 Members of pharmacy and
therapeutic committees report a need for more education on
RWE to support reimbursement decisions.34 Alwardt35 argue
that “with greater perspective, payers.can make more
informed decisions about access and formulary inclusion,
which will help improve patient outcomes..” It is likely that
the influence of RWE in both policy and payor decisions will
increase as its use in research and regulatory decision-making
continues to expand, particularly as the value of RCT and
RWE findings seen together is increasingly recognized.36
Challenges Associated With RWD
RWD offer the potential for tremendous increases in the
breadth, depth, and diversity of our understanding in many
areas of medicine,37 but the volume, velocity, and variety
with which they come pose unique challenges.
Data Quality: Variety and Veracity
Those hesitant to use or accept RWD in research often
cite concerns about data quality.2,5,23,38 Data collected in the
real-world setting do not always have the same quality con-
trols as those collected in a clinical trial setting.23 For exam-
ple, EHR structures and content are not identical across
institutions.18 Agencies such as the EMA and FDA have
published recommendations to improve the quality of RWE
including the need to define standard formats for data sets,
protocols, and tools to ensure reproducibility. FDA guidelines
include suggestions for data accrual, data assurance and qual-
ity control, and a recommendation for a priori study protocols
and analysis plans to avoid potential bias. Following these
recommendations “optimizes the reliability, quality, and use-
fulness of the data.”3,23,39
Examples of solutions to quality concerns include the
publication of guidelines and best practices for data creation,
collection, and analysis and the recent adoption of universal
device identifiers which make it easier to capture device
details in AE reporting.3,5,18,23,39,40 Tools to assist researchers
in determining quality and appropriateness of a RWD data set
for a particular research question or purpose have also been
published.3,6,16,23,32,39–41 Methods to account for selection
bias, handle missing data, and other potential biases are
mature and constantly under development and improvement,
enabling researchers to draw valid conclusions for RWE.42,43
Privacy
The increased use of RWE has sparked increased
interest in patient privacy. The real potential for security
breaches, the disclosure of sensitive health information, and
the fear of its use for discriminatory purposes are of common
concern.44–46 The use of RWD for evaluation or performance
monitoring has sometimes been described as “big brother,”
specifically with respect to monitoring of physician and hos-
pital performance where results are published and could affect
patient choice and profits.27,47,48 In such cases, the concern of
data quality or the appropriateness of the analytical approach
also come into discussion.46 Solutions to privacy issues
include scrupulous attention to data security and range from
simple deidentification of data and suppression of small cells
www.jorthotrauma.com | S3
Zura et al J Orthop Trauma  Volume 35, Number 3 Supplement, March 2021
in published work to the development of federated learning
models and platforms to allow for analyses to be conducted
on encrypted data.46,49,50 Appropriate informed consent pro-
cedures can also help mitigate privacy concerns.46
Transparency
Efforts to improve data privacy are sometimes at odds
with the need for data transparency.46 Creating linkages
between some RWD sources (whether it be a hospital,
insurer, researcher, or government agency) can sometimes
be difficult because of privacy and confidentiality concerns,
thus limiting global transparency.51–53 Policies on data man-
agement and procedures that focus on transparency and
accountability can help ensure compliance and prevent or
alleviate related challenges.54 Public posting of queries, pro-
tocols, and results are also best practices for improving trans-
parency, as well as promoting replicability or
reproducibility.37
CONCLUSIONS
RWE, which is increasingly available and accessible,
can further our understanding of disease, disease progression,
and effective, safe treatments with the speed and accuracy
required by the health care environment and patients today.
RWD methods are mature, and they have a strong track
record of utility. The role and value of RWE in research,
practice, and policy continues to grow. Innovations reached
thus far, including solutions to mitigate challenges, are
evidence that the more RWE is used, and the more it is
understood, the more useful it will be. The expanded
acceptance of RWE to support regulatory decision-making
ensures that its use will only continue to increase, particularly
as researchers find ways to use RCT and RWE studies
together to strengthen evidence to support the use of drugs
and devices.
ACKNOWLEDGMENTS
The authors thank Irene Shupe and Nicole Sgherza for
their support in the organization of this article.
REFERENCES
1. Sanchez-Pinto LN, Luo Y, Churpek MM. Big data and data science in
critical care. Chest 2018;154:1239–1248.
2. US Food and Drug Administration (FDA). Framework for FDA’s Real-
World Evidence Program [www.fda.gov]. 2018. Available at www.fda.
gov/media/120060/download/. Accessed January 25, 2020.
3. Berger M, Daniel G, Frank K, et al. A Framework for Regulatory Use of
Real-World Evidence [www.healthpolicy.duke.edu]. 2017. Available at:
https://healthpolicy.duke.edu/publications/framework-regulatory-use-
real-world-evidence. Accessed February 14, 2020.
4. Trifiro G, Sultana J, Bate A. From big data to smart data for pharmaco-
vigilance: the role of healthcare databases and other emerging sources.
Drug Saf. 2018;41:143–149.
5. Sherman RE, Anderson SA, Dal Pal GJ, et al. Real-world evidence- what
is it and what can it tell us? N Engl J Med. 2016;375:2293–2297.
6. Dreyer N. Advancing a framework for regulatory use of real-world evi-
dence: when real is reliable. Ther Innov Reg Sci. 2018;52:362–368.
7. Lamberti MJ, Kubick W, Awatin J, et al. The use of real-world evidence
and data in clinical research and postapproval safety studies. Ther Innov
Reg Sci. 2018;52:778–783.
S4 | www.jorthotrauma.com
Copyright © 2021 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
8. Ray WA, Griffin MR, Schaffner W, et al. Psychotropic drug use and the
risk of hip fracture. N Engl J Med. 1987;316:363–369.
9. Lin KB, Yang YP, Lee YH, et al. The incidence and factors of hip
fractures and subsequent morbidity in Taiwan: an 11-year population-
based cohort study. PLoS One. 2018;15:e0192388.
10. Austin PC. An introduction to propensity score methods for reducing the
effects of confounding in observational studies. Multivariate Behav Res.
2011;48:399–424.
11. Oichi T, Oshima Y, Chikuda H, et al. In-hospital complication rate
following microendoscopic versus open lumbar laminectomy: a propen-
sity score-matched analysis. Spine J. 2018;18:1815–1821.
12. Hariri RH, Fredericks EM, Bowers KM. Uncertainty in big data analyt-
ics: survey, opportunities, and challenges. J Big Data. 2019;6:44.
13. Pedoia V, Haefeli J, Morioka K, et al. MRI and biomechanics multidimen-
sional data analysis reveals R2-R1r as an early predictor of cartilage lesion
progression in knee osteoarthritis. J Magn Reson Imag. 2018;47:78–90.
14. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development and
validation of a risk stratification tool for ward patients. Am J Respir Crit
Care Med. 2014;190:649–655.
15. Goldshtein I, Gerber Y, Ish-Shalom S, et al. Fracture risk assessment
with FRAX using real-world data in a population-based cohort from
Israel. Am J Epidemiol. 2017;187:94–102.
16. Deo RC. Machine learning in medicine. Circulation. 2015;132:1920–
1930.
17. Cabitza F, Locoro A, Banfi G. Machine learning in orthopedics: a liter-
ature review. Front Bioeng Biotechnol. 2018;6:75.
18. Callahan A, Fries JA, Ré C, et al. Medical device surveillance with
electronic health records. NPJ Digit Med. 2019;2:94.
19. Andre EB, Reynolds R, Caubel P, et al. Trial designs using real-world
data: the changing landscape of the regulatory approval process.
Pharmacoepidemiol Drug Saf. 2019:1–12.
20. Bate A, Hornbuckle K, Juhaeri J, et al. Hypothesis-free signal detection
in healthcare databases: finding its value for pharmacovigilance. Ther
Adv Drug Saf. 2019;10:1–9.
21. Dasa V, Lim S, Heeckt P. Real-world evidence for safety and effectiveness of
repeated courses of hyaluronic acid injections on the time to knee replace-
ment surgery. Am J Orthop. 2018;47. doi: 10.12788/ajo.2018.0058.
22. Ball R, Robb M, Anderson SA, et al. The FDA’s sentinel initiative- a
comprehensive approach to medical product surveillance. Clin
Pharmacol Ther. 2016;99:265–268.
23. US Food and Drug Administration (FDA). Use of Real-World Evidence
to Support Regulatory Decision-Making for Medical Devices [www.fda.
gov]. 2017. Available at: https://www.fda.gov/media/99447/download.
Accessed January 25, 2020.
24. Rubenfeld GD. Using computerized medical databases to measure and to
improve the quality of intensive care. J Crit Care. 2004;19:248–256.
25. Abrams GD, Greenberg DR, Dragoo JL, et al. Quality measures in
orthopaedic sports medicine: a systematic review. Arthroscopy. 2017;
33:1896–1910.
26. Curtis JR, Sathitratanacheewin S, Starks H, et al. Using electronic health
records for quality measurement and accountability in care of the seri-
ously ill: opportunities and challenges. J Palliat Med. 2018;21:S52–S60.
27. Naunheim KS, Sabik JF. Big brother is watching. Ann Thorac Surg.
2017;104:1768–1769.
28. Mack C, Christian J, Brinkley E, et al. When context is hard to come by:
external comparators and how to use them. Ther Innov Reg Sci. 2019:
2168479019878672.
29. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375:454–463.
30. Goring S, Taylor A, Müller K, et al. Characteristics of non-randomised studies
using comparisons with external controls submitted for regulatory approval in
the USA and Europe: a systematic review. BMJ Open. 2019;9:e024895.
31. Pfizer. US FDA Approves Ibrance (Palbociclib) for the Treatment of Men
with HR+, HER2-Metastatic Breast Cancer [www.pfizer.Com]. 2019.
Available at https://www.pfizer.com/news/press-release/press-release-
detail/u_s_fda_approves_ibrance_palbociclib_for_the_ treatment_of_
men_with_hr_her2_metastatic_breast_cancer. Accessed February 2,
2020.
32. Caliebe A, Leverkus F, Antes G, et al. Does big data require a methodolog-
ical change in medical research?. BMC Med Res Methodol. 2019;19:125.
33. Mahlmann L, Reumann M, Evangelatos N, et al. Big data for public
health policy-making: policy empowerment. Public Health Genomics.
2017;20:312–320.
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
J Orthop Trauma  Volume 35, Number 3 Supplement, March 2021
34. Malone DC, Brown M, Hurwitz JT, et al. Real-World Evidence: useful in
the real world of US payer decision making? How? When? And what
studies. Value Health. 2018;21:326–333.
35. Alwardt S. Rethinking the Use of Real-World Evidence to Improve Policy
Decisions [www.ajmc.com]. 2019. Available at: https://www.ajmc.
com/contributor/sarah-alwardt-phd/2019/04/rethinking-the-use-of-realworld-
evidence-to-improve-policy-decisions. Accessed October 19, 2020.
36. Dreyer NA. Shaping the Future of Real-World Evidence [www.pharma-
voice.com]. 2020. Available at: https://www.pharmavoice.com/digital-
edition/july-august-2020/#148. Accessed October 21, 2020.
37. Martin D. FDA Approaches to Analytical Challenges Posed by Big Data
[www.ema.europa.eu]. 2015. Available at: https://www.ema.europa.
eu/en/documents/presentation/presentation-fda-approaches-analytical-
challenges-posed-big-data-david-martin_en.pdf. Accessed January 25, 2020.
38. Belle A, Thiagarajan R, Soroushmehr R, et al. Big data analytics in
healthcare. Biomed Res Int. 2015;2015:370194.
39. Mahendraratnam N, Silcox C, Mercon K, et al. Determining Real-
World Data’s Fitness for Use and the Role of Reliability [www.
healthpolicy.duke.edu]. 2019. Available at: https://healthpolicy.
duke.edu/publications/determining-real-world-datas-fitness-use-
and-role-reliability. Accessed February 14, 2020.
40. Girman CJ, Ritchey ME, Zhou W, et al. Considerations in characterizing
real-world data relevance and quality for regulatory purposes: a commen-
tary. Pharmacoepidemiol Drug Saf. 2018:1–4.
41. Ritchey MB, Girman CJ. Evaluating the feasibility of electronic health
records and claims data sources for specific research purposes. Ther
Innov Regul Sci. 2020;54:1296–1302.
42. Faria R, Hernandez Alava M, Manca A, et al. NICE DSU Technical
Support Document 17: The Use of Observational Data to Inform
Estimates of Treatment Effectiveness for Technology Appraisal:
Methods for Comparative Individual Patient Data [www.nicedsu.org.uk].
2015. Available at: http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD17-
DSU-Observational-data-FINAL.pdf. Accessed February 17, 2020.
43. Schmidt AF, Klungel OH, Groenwold RH, et al. Adjusting for confound-
ing in early postlaunch settings: going beyond logistic regression models.
Epidemiology. 2016;27:133–142.
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2021 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
A Primer on Real-World Evidence
44. Kulynych J, Greely HT. Clinical genomics, big data, and electronic
medical records: reconciling patient rights with research when privacy
and science collide. J Law Biosci. 2017;4:94–132.
45. Mittelstadt BD, Floridi L. The ethics of big data: current and foreseeable
issues in biomedical contexts. Sci Eng Ethics. 2016;22:303–341.
46. Currie J. “Big data” versus “big brother”: on the appropriate use of large-
scale data collections in pediatrics. Pediatrics 2013;131(suppl 2):S127–S132.
47. Centers for Medicare and Medicaid Services (CMS). CMS National Dry
Run: All-Cause Unplanned Readmission Measure for 30 Days Post
Discharge from Inpatient Rehabilitation Facilities 2015 [www.cms.
gov]. 2015. Available at: https://www.cms.gov/Medicare/Quality-
Initiatives-Patient-Assessment-Instruments/IRF-Quality-Reporting/Downloads/
IRF-SODF-Presentation-CMS-National-Dry-Run-October-20-2015-edit-11-15.
pdf. Accessed February 2, 2020.
48. Mittal M, Wang CH, Goben AH, et al. Proprietary management and
higher readmission rates: a correlation. PLoS One. 2018;13:e0204272.
49. Tu S, Kaashoek MF, Madden S, et al. Processing analytical queries over
encrypted data. Proc VLDB Endow. 2013;6:1–12.
50. Lu S, Zhang Y, Wang Y, et al. Learn electronic health records by fully
decentralized federated learning, in Proceedings of Advances in Neural
Information Processing Systems Workshop on Federated Learning for
Data Privacy and Confidentiality in Conjunction. Conference on Neural
Information Processing Systems; December 8–14, 2019; Vancouver,
Canada.
51. Langan SM, Schmidt SA, Wing K, et al. The reporting of studies con-
ducted using observational routinely collected health data statement for
pharmacoepidemiology (RECORD-PE). BMJ. 2018;363:k3532.
52. Pratt N, Mack C, Meyer AM, et al. Data linkage in pharmacoepidemi-
ology: a call for rigorous evaluation and reporting. Pharmacoepidemiol
Drug Saf. 2020;29:9–17.
53. Benchimol EI, Smeeth L, Guttmann A, et al. The reporting of studies
conducted using observational routinely-collected health data
(RECORD) statement. PLoS Med. 2015;12:e1001885.
54. Vayena E, Dzenowagis J, Brownstein JS, et al. Policy implications of
big data in the health sector. Bull World Health Organ. 2018;96:66–
68.
www.jorthotrauma.com | S5