RBM Metrics Report - December 2019

MEASURING
THE IMPACT
OF RISK BASED
MONITORING
The past, present and
future of RBM metrics
The Risk Based Monitoring (RBM) Initiative was
founded in 2012 in response to regulatory guidance
encouraging the adoption of risk-based monitoring
to improve sponsor oversight of clinical studies and
to ensure patient safety and clinical trial data quality.
The focus of this response was the development of a
methodology to enable organizations to successfully
deploy and scale RBM. Since its inception, the Risk
Based Monitoring Initiative has matured and the
industry in general has become increasingly familiar
with RBM concepts and with the TransCelerate RBM
methodology. Similarly, measuring the impact of
adopting RBM on critical stakeholders (patients, sites,
regulatory authorities and sponsors) has also evolved.
In light of the organic growth and development of RBM,

this document aims to provide the following:
An overview of the TransCelerate RBM initiative’s

original vision for specific metrics with the potential to
effectively measure the value and impact of RBM from
multiple perspectives.
An analysis of how those original, suggested metrics

have been put into realworld practice.
TABLE OF CONTENTS
PAST:
Risk Based Monitoring Metrics Through The Years
» A Review of the Risk Based Monitoring (RBM) Initiative
» Concept and Vision for Metrics
» Lessons Learned
PRESENT:
Risk Based Monitoring Metrics Today
» Best Practices for Defining and Measuring RBM Value through Metrics
» New Methods to Measure the Impact of RBM
FUTURE:
Risk Based Monitoring Metrics Evolution
» Preparing for Health Authorities
» Trends and Evolution
APPENDIX
» Limitations
» Metrics Defined and Expected Observations
» New Key Performance Indicators
TIMELINE OF RISK BASED MONITORING INITIATIVE
2012-2017 2018-2019 2020 & Beyond

PAST PRESENT FUTURE
Active Phase: Deliver & Measure Mature Phase: Facilitate Adoption Next Phase: RBM Evolution
2012 2019 2020 on…

TransCelerate TransCelerate RBM continues
forms Metrics Survey to evolve
One of its first initiatives is 18 member companies are Multiple factors including
RBM, in response to FDA invited to participate in an ICH E6, E8 revisions, evolving
guidance. anonymous survey to better quality management
» 8 Methodology understand how the value systems, and technology
Framework Papers of RBM implementation is advances have the potential
defined and assessed. to impact RBM.
» Multiple Supporting » Focus on Facilitating
Tools and Resources for Awareness, Adoption and
Industry Use (Assets) Best Practice Sharing
» TransCelerate collects, » Ongoing High Visibility

anonymizes and Engagements across
aggregates RBM metrics Industry (Healthcare
data quarterly from Authorities, Conferences,
member companies. Site Advocacy Groups,
Implementation increases Contract Research
most significantly from Organizations)
2015-2016
» Perception is that RBM
is improving quality
and efficiency, however
quantifying the magnitude
of improvement remains
challenging
4 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

PAST
COLLECTING RBM DATA QUARTERLY

Methods 1
Data was collected from participating member companies quarterly from 2013-2017,
resulting in >1000 observations.
Guided by a defined set of RBM metrics, member companies voluntarily reported data to
TransCelerate on clinical trials where they implemented RBM.
Member companies were asked to rate change over time for each metric as “better,”
“worse” or “about the same” (as compared to each company's internal baseline
expectation).
Data was blinded, aggregated and reported by TransCelerate each quarter.
We have further aggregated the data collected from 2013-2017 to analyze RBM trends
and produce cumulative observations.
Metrics 2
Click the
for more details.
Average number Percentage of Number of Average Monitoring

of major/critical unreported, Significant Protocol (all types)
audit findings confirmed SAEs as Deviations
compared to total
SAEs as discovered
through any method
Average interval Median number Median number Median number

between on-site of days from issue of days from of days from query
monitoring visits open to close patient visit to open to close
eCRF data entry
1 For more on reporting limitation, see Limitations in the appendix

2 Note that the metrics above may be aligned to different categories,
depending on the organization and context.

PAST
CUMULATIVE DATA OBSERVATIONS (RELATIVE %)

Participating Member Companies voluntarily reported data quarterly
from 2013-2017 about their clinical trials implementing RBM.
Not all Member Companies provided data for all metrics each quarter.
Worse Better or Same
Audit Findings 8% 62% Click the

30 % for more details.
SAE Reporting 16% 53%

Worse Better or Same 31%
Significant Protocol
27% 53%
Deviations
Audit Findings 8% 19% 62%
30%
Overall Monitoring Cost 20% 59%
SAE Reporting 16% 21%
53%
31 %
Significant Protocol
On-Site Visit Interval
27% 19% 53% 56%
Deviations 19% 25%
Issue Open to Close

Overall Monitoring Cost 20% 17% 59% 59%
24%
21 %
eCRFInterval
On-Site Visit Entry 19% 22% 56% 54%
24%
25%
Better
Query
Issue Open Open to Close
to Close 17% 18% 59% 55%
27%
Same
24%
Worse
eCRF Entry 22% 54%
24%
Better
Query Open to Close 18% 55%
27%
Same
HIGHLIGHTS: Worse
The data shows the benefit of RBM implementation revealed

in >50% contribution of improvement (“better” signals) in all
collected metrics.
All 8 metrics had results of >50% “better” and >70% “better”

or “about the same”
1 Results Aligned with Expected Observations, see appendix

2 Results Aligned with Potential Observations, see appendix

PAST
DISCUSSION
2013-2017 TRENDS IN RBM IMPLEMENTATION
Implementation of RBM appears to improve the quality, timeliness and efficiency of
clinical trials as shown in all metrics, which are all indicators of potentially improved
patient safety monitoring and improved site operation quality and efficiency, all factors
critical to health authorities. The metrics suggest that sites are able to spend more time
on issues that matter, improving patient safety. Improvements in timeliness and efficiency
may improve cycle times, bringing medications to patients faster.
In the majority of cases, the expected outcomes appear to have been realized and, in
some cases, the potential expectations have also been realized.
In general, RBM Implementation:

• Allows for the increase in quality of clinical trials (measured as audit findings, issue
resolution time, protocol deviations). Quality should increase over the maturation
of the trial (alignment with potential expectations since we are looking for areas of
risk earlier). The increased quality of clinical trials indicates an enhanced focus on
patient safety and operational efficiency, a focus that is a priority for subjects, sites,
regulators and sponsors.
• Allows sponsors to more efficiently target monitoring resources to the sites that
require greater focus (better signals for the metrics overall monitoring cost and on
site visit interval). More upfront risk assessment and planning positively influence
study design, allowing sites to focus on patient safety and other critical operations.
• Is most effective when using centralized approaches to monitor all of the metrics
and risks, as centralized monitoring is associated with early identification and
mitigation of data quality risk/issue(s) plays a role in supporting better monitoring of
patient safety and enhancing subject protection.

PRESENT
UNDERSTANDING CURRENT STATE

What does RBM look like today?
In 2019, TransCelerate conducted an anonymous survey of member companies to
understand the organizations’ own applications of the original RBM metrics developed
in 2013.
Our objective was to analyze current member company use of the original metrics,
assess how the benefits of RBM are currently measured and determine whether
new metrics have been developed to further successfully define the value of RBM
implementation. The scope of the survey was to determine how companies are
measuring the value/benefits of RBM implementation, not to measure trial operations or
site performance.
The following analysis represents an interpretation of the experiences of TransCelerate

member companies and should not be considered a “one-size-fits-all” implementation
guidance.
• 18 member companies responded to the survey.
• Results were blinded, aggregated and reported by a third party,
then analyzed via collaboration across membership.
2019 Member Company Survey Results

Are the original metrics being used to evaluate the implementation of RBM? 1
Currently being used by the highest Have never been used by

number of member companies most companies
Average number of major/critical Percentage of unreported,

audit findings confirmed SAEs as compared
to total SAEs as discovered
Number of Significant Protocol through any method
Deviations
Average Monitoring
Median number of days from (all types) cost
issue open to close
Average interval between
Median number of days from on-site monitoring visit
patient visit to eCRF data entry
Median number of days from

query open to close
1 Results listed from greatest used to least used

PRESENT
2019 Member Company Survey Results continued
Key Decision Drivers Behind RBM Adoption
Both Quality/Compliance and Number of responses 1
Efficiency/Cost are primary 10

9
drivers for implementing an
8
RBM model. 7
6
Member companies indicated
4
that RBM adoption began as a
project in pursuit of improved 2
1 1
Quality/Compliance. They also
0
indicated that RBM's potential to Quality/ Efficiency/ Scientific Other*
Compliance Cost Validity/ *Both Quality/Compliance and
lead to efficiency gains for both Robustness of the Scientific Valdity/Robustness of
Risk Assessment the Risk Assessment Discussions
sponsors and sites/CROs was Discussions
a critical factor in deciding to

implement RBM.
Types of Trials in Scope of RBM
Implementation of RBM is Number of Member Companies
now seen across all phases of 15

14
Interventional Clinical Trials.
12
14 companies report utilizing
RBM in all phases of their studies
9
currently. The others have taken
various, staggered approaches.
6
Member companies approaches
to implementation have evolved 3
2
over time to expand to current 1 1
state. 0
All Clinical Phase I/II/III Phase II/III None
Trials Only Only (RBM not
implemented)
1 Note that survey respondents were only allowed one response

to the question. It is likely that both Quality/Compliance and
Efficiency/Cost were organizational goals for RBM implementation.

PRESENT
2019 Member Company Survey Results continued
Components Implemented with RBM
Risk assessments, reduced Number of responses 2
Site Data Verification (SDV) 20
and a more remote approach 18

to analysis and monitoring 16 16
are the major components 15
14
consistently seen across 13
companies who have adopted
12
a RBM model (See Papers and 10
Tools for a comprehensive list).
7 7
The risk based approach
appears to be spreading to 5
other, related areas (e.g., data 3

cleaning).
0
Trial Targeted Off-site Centralized Statistical On-site Trial level Data Cleaning Other
Level Risk SDV Monitoring/ Monitoring Monitoring/ Monitoring Quality Determined
Assessment Remote Surveillance Determined Tolerance by Data
Monitoring by Site Risk Limits Criticality
Levels (QTLs)
How long did it take to see/measure the impact(s) of RBM implementation?
Whilst it appears that RBM Number of responses 1
contributed positively to the 6

6
development process within
4
1-3 years of implementation, 4
3 3
few companies are tangibly 2
2
measuring its direct impact
0
through metric analysis, with Not 2-3 1-2 I Don’t 6 months-
Applicable years years Know 1 year
many still developing their
metrics and collecting data.
While quality is a key decision

driver for implementation,
organizations struggle to
measure quality directly,
and note significant change
management efforts are
necessary to support the
process. 1 Participants could select only one response
2 Participants could select more than one response

PRESENT
RISK BASED MONITORING

How are companies evaluating value?
Selections from the 2019 Member Company Survey:
“The benefit seen so far is… ‘knowing Realized Value

data better,’ more smooth data base TransCelerate Member
lock etc.” Companies have already
provided qualitative
information.
“Our biggest challenges going forward,

as we evolve the model—how do “[RBM] has allowed us
to improve patient safety
we objectively establish that the
and data integrity.”
comparative quality of the trial data
—Member Company, 2017
that is submitted in our filings is as
good if not better?”
“With real time
monitoring, we were
able to identify instances
of non-compliance and
“[The value of RBM] could be value eventually amend the
of the process, value of the people, protocol to make the
and value of a tool. This is likely both language stronger.”
quantitative and qualitative with —Member Company, 2017
different value stories for the different
audiences (study team, functions, “[RBM] has enabled
leadership, sites, regulators).” strong cross-functional
and geographical
collaboration among
medical and safety staff.”
—Member Company, 2017

FUTURE
WHAT ARE THE CURRENT CHALLENGES FOR RBM

METRICS?
The existing measurement process includes challenges that prevent us from quantifying
the benefits of RBM accurately.
• Many Member Companies indicate they perceive the benefits of RBM
implementation, yet less than half use historical controls /baseline data to
demonstrate the value of RBM implementation.
• Measuring RBM value for sponsors is not necessarily one-size-fits-all. The same
metrics are perceived differently across sponsor companies. Similarly, as RBM
models differ among Member Companies, their approach to measuring RBM value
also differs.
• Multiple changes are introduced throughout study operations in parallel (e.g., new
systems, roles, requirements and/or monitoring strategies), making it difficult to
understand the true impact of RBM on each trial.
• Limitations to being able to consistently define, measure and benchmark quality in a
scalable manner create challenges to quantifying the value of RBM and other quality-
focused initiatives.
HOW ARE RBM METRICS EVOLVING?

The art and science of measuring and demonstrating the benefits of RBM
are still evolving.
The original RBM metrics were proxy metrics that measured for quality, but did not
always directly measure the impact attributable to RBM.
While half of the Member Companies have continued using these original metrics, several
have defined new/novel Key Performance Indicators (KPIs) to measure their perceived
value of RBM. Roughly 25% of these new KPIs are classified as “extremely useful.”
Best practices appear to include defining measures tailor-made to the chosen RBM
model and the individual organization’s operational design. This indicates that the
identification of optimal metrics to quantify the value of RBM is still a work in progress
across the member companies.
Some metrics were found by member companies to be only somewhat useful since they
are proxies for improvement, and there are many other initiatives to improve quality, so
they are difficult to attribute solely to RBM implementation.

FUTURE
WHAT’S NEXT FOR RISK BASED MONITORING?

As the industry continues to adapt to risk-based approaches for clinical development,
and as technology evolves to support them, RBM is expected to continue to evolve.
Quality Tolerance Limits are “the next frontier” for RBM and future research should
study how to effectively implement them. The RBM team had published implementation
considerations for establishing quality tolerance limits (QTLs) and risk reporting in the
clinical study report (CSR) in 2017 that is available here. TransCelerate is planning to
publish additional QTL guidance and best practices in the future.
Interest in analyzing the results of RBM-specific regulatory inspections has increased.

While TransCelerate has collected some initial data around this topic, insights remain
limited.
In addition, the interest in more in-depth insights into RBM’s impact on patients and sites
is an opportunity for further research.
Overall, RBM has evolved tremendously over several years and that growth is expected
to continue. We welcome direct feedback from Health Authorities and look forward to
the innovative ways in which all companies continue to incorporate RBM methodologies
into their own processes and assess the value of their implementation.

Measuringthe Impact of Risk Based Monitoring | The past, present and future of RBM Metrics APPENDIX
LIMITATIONS
The authors acknowledge the following limitations of this work 1
No Non-numerical Observations are

Control Group Assessment Trend-based
Lack of a control group Change over time was This commentary is based
for comparison (Non- measured using a non- on observed trends, as
RBM studies) makes the numerical assessment. there is not sufficient
interpretation of the data Member companies data to draw robust
less complete. However, assigned a value to each conclusions. Quarterly
it should be noted that metric using “better, data collection ended in
many member companies worse or about the 2017 and the number of
no longer have control same” as compared to trials reported with long
groups, as RBM has been each company's internal maturities (greater than 1-2
fully-embedded across baseline expectation. years) is limited. However,
clinical trial portfolios. This measurement does the team has aggregated
not allow for precise all data available to
comparisons, as each analyze potential trends
member company defined and observations.
relative values differently.
Go back to Collecting Quarterly RBM Data
1 Not all companies reported data each quarter. Over the course of the reporting
periods, the number of companies reporting data ranged from 4 to 11.

METRICS AND EXPECTED OBSERVATIONS

The 8 original metrics below were developed in 2013 to measure the
impact of RBM and guided the collection of quarterly data from member
companies from 2013-2017.
Metric Expected Potential/Alternative

Developed in 2013 Observations Observations
Average number of Average number of major/ Audit findings may initially rise
major/critical audit critical findings per audited due to focus on critical data and
findings site will decrease processes
Percentage of Unreported, confirmed Percentage of unreported,

unreported, confirmed SAEs will decrease confirmed SAE findings may rise
SAEs as compared to initially due to shift in focus from
total SAEs as discovered SDV to SDR
through any method
Number of Significant Significant Protocol Significant protocol deviation

Protocol Deviations Deviations will decrease findings may rise initially due to
shift in focus from SDV to SDR
Average Monitoring Average monitoring costs Costs may remain flat until second
(all types) cost will decrease quarter of analysis or later
Average interval Interval between on-site Average interval between on-site

between on-site monitoring visits will monitoring visits may remain flat
monitoring visits increase until second quarter of analysis or
later
Median number of days Median number of days Findings initially may rise if issues
from issue open to close from issue open to close will management process is new to
decrease the organization
Median number of days There are no expectations The site may delay performing a
from patient visit to to improve the median crucial function that empowers
eCRF data entry number of days from patient central monitoring due to the
visit to eCRF data entry potential decrease in on-site visits
Median number of There are no expectations The site may delay performing
days from query open to improve for the median a crucial function due to the
to close number of days from query potential decrease in on-site visits
open to close
Go back to Cumulative Data Observations

KEY PERFORMANCE INDICATORS (KPIs)

Examples 1 of new Key Performance Indicators for RBM developed by
Member Companies. The 9 Member Companies that responded provided
28 new/novel KPIs they then also ranked by usefulness.
Rank Process Compliance Data
Extremely • Time from data "cut" to Action 3 • Important • Missed

Useful • Ratio of on-site to off-site monitoring Protocol Assessments2
visits 4 Deviation • Dosing Deviation
Incidence 3 Incidence 3
• SDR & SDV backlog 3
• % queries resolved in 7 days 3

• % pages submitted in 7 days 3
Somewhat • Reports for centralized monitoring— • TMF • SAE/AE Rates2
Useful user statistics to indicate frequency and Compliance 2 • Query rate
duration of use 3 (per 1000 data
• Ratio of Data correction XX days after points) 3
initial data entry3
• RBM user satisfaction survey3
• Query rates2
• Qualitative interviews with HQ trial teams2
• Survey for use and usefulness of site risk
indicator report 2
Neutral • Action item aging 3 • CAPAs close on • Query aging 3
• External data review status 2 time (site) 3 • Ratio of
• CAPA # overall 3 number of AE
• eTMF status2 emerging per
subject 4
• Ratio of number
of AE emerging
per subject 4
Not very • On-site vs. remote visit ratio: Ratio • Ratio of • Ratio of
Useful calculated as number on-site to remote Missing data Missing data
visits 4 for the primary for the primary
endpoint 4 endpoint 4
Go back to How are RBM Metrics Evolving
1 These are the direct answers from survey respondents, we have not summarized, word-smithed, or tried to
categorize them. Some information may be contradictory since it was submitted by different companies.
2 Qualitative Indicator
3 Quantitative Indicator
4 Duplicate

PAPERS AND TOOLS
Statistical Monitoring/Surveillance
Statistical Monitoring in Clinical Trials: Best Practices for Detecting Data Anomalies
Suggestive of Fabrication or Misconduct
Centralised Monitoring
Defining a Central Monitoring Capability: Sharing the Experience of TransCelerate
BioPharma’s Approach, Part I
Defining a Central Monitoring Capability: Sharing the Experience of TransCelerate

BioPharma’s Approach, Part 2
Off-Site/Remote Monitoring
See Table 2 in the RBM Methodology Position Paper
On-Site Monitoring determined by Site Risk Levels

Site Level Risk Assessment Considerations
Trial Level Risk Assessment

TransCelerate Risk Assessment and Categorization Tool (RACT)
Quality Tolerance Limits

Risk-Based Quality Management: Quality Tolerance Limits and Risk Reporting
Go back to Components Implemented with RBM

RBM Metrics Report - December 2019

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

RBM Metrics Report - December 2019

Uploaded by

Copyright:

Available Formats

MEASURING

In light of the organic growth and development of RBM,

An overview of the TransCelerate RBM initiative’s

An analysis of how those original, suggested metrics

2012-2017 2018-2019 2020 & Beyond

2012 2019 2020 on…

» TransCelerate collects, » Ongoing High Visibility

4 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

COLLECTING RBM DATA QUARTERLY

Data was blinded, aggregated and reported by TransCelerate each quarter.

Average number Percentage of Number of Average Monitoring

Average interval Median number Median number Median number

1 For more on reporting limitation, see Limitations in the appendix

5 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

CUMULATIVE DATA OBSERVATIONS (RELATIVE %)

Worse Better or Same

Audit Findings 8% 62% Click the

SAE Reporting 16% 53%

Issue Open to Close

The data shows the benefit of RBM implementation revealed

All 8 metrics had results of >50% “better” and >70% “better”

1 Results Aligned with Expected Observations, see appendix

6 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

In general, RBM Implementation:

7 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

UNDERSTANDING CURRENT STATE

The following analysis represents an interpretation of the experiences of TransCelerate

2019 Member Company Survey Results

Currently being used by the highest Have never been used by

Average number of major/critical Percentage of unreported,

Median number of days from

1 Results listed from greatest used to least used

8 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

2019 Member Company Survey Results continued

Key Decision Drivers Behind RBM Adoption

Both Quality/Compliance and Number of responses 1

Efficiency/Cost are primary 10

a critical factor in deciding to

Types of Trials in Scope of RBM

Implementation of RBM is Number of Member Companies

now seen across all phases of 15

1 Note that survey respondents were only allowed one response

9 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

2019 Member Company Survey Results continued

Components Implemented with RBM

Risk assessments, reduced Number of responses 2

Site Data Verification (SDV) 20

and a more remote approach 18

other, related areas (e.g., data 3

How long did it take to see/measure the impact(s) of RBM implementation?

Whilst it appears that RBM Number of responses 1

contributed positively to the 6

While quality is a key decision

10 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

RISK BASED MONITORING

“The benefit seen so far is… ‘knowing Realized Value

“Our biggest challenges going forward,

11 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

WHAT ARE THE CURRENT CHALLENGES FOR RBM

HOW ARE RBM METRICS EVOLVING?

12 Copyright ©2019 TransCelerate BioPharma Inc. All rights reserved.

WHAT’S NEXT FOR RISK BASED MONITORING?

Interest in analyzing the results of RBM-specific regulatory inspections has increased.