Symptom Science and Palliative Care

Integrating Patient-Reported Outcomes Into the Care of

People With Advanced Cancer—A Practical Guide
Julia Lai-Kwon, MBBS, BMedSci, MPH1,2; Elissa Thorner, MHS3; Claudia Rutherford, PhD4; Norah Crossnohere, PhD5; and
Michael Brundage, MSc, FRCPC, MD6

DOI https://doi.org/10.1200/EDBK_438512

Accepted April 6, 2024

OVERVIEW
Published May 24, 2024

Patient-reported outcomes (PROs) are being increasingly integrated into routine clinical Am Soc Clin Oncol Educ Book
practice to enhance individual patient care. This has been driven by recognition of the benefits 44:e438512
of PROs in enhancing symptom management, patient satisfaction, quality of life, and overall © 2024 by American Society of
survival, and reductions in acute health care utilization. These benefits are reflected in the Clinical Oncology
emergence of value-based health care initiatives incorporating PRO symptom monitoring such
as the Enhancing Oncology Model in the United States. However, implementing PROs can be
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145

challenging and it can be difficult to know where to begin to select appropriate PROs, and
effectively display and appropriately interpret PRO data. This manuscript summarizes an
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.

educational session at the 2024 ASCO Annual Meeting, which provided practical guidance to
clinicians seeking to incorporate PROs into the care of people with advanced cancer. We focus
on why it is important to collect PROs in routine care from a patient’s perspective, how to select
PROs for symptom monitoring (including using static patient-reported outcome measures and
newer item libraries), and highlight key pearls and pitfalls in the display and interpretation of
PROs. We highlight the breadth of existing resources available to guide clinicians in PRO
implementation.

INTRODUCTION the person to adhere to treatment.2 Tolerability is a complex

This paper summarizes an educational session titled, “Inte-
grating Patient-Reported Outcomes and Tolerability Assessment”
in the everyday care of patients with advanced cancer pre-
sented at the 2024 ASCO Annual Meeting. The objectives of
this session were to (1) illustrate the importance of patient-
reported outcomes (PROs) from the patient’s perspective; (2)
review the benefits of using PROs in clinical practice, focusing
on patient tolerance of treatments in the advanced cancer
setting; (3) review how to select PROs for symptom moni-
toring; (4) highlight common pearls and pitfalls in the in-
terpretation of PROs; and (5) summarize available tools for
successful implementation of PROs in the clinic.
PROs are people’s own reports of how they feel (eg, symptom
scores), function (eg, physical functioning), live their lives
(eg, social functioning), and survive (eg, global quality of life).
These outcomes are assessed with standardized validated
questionnaires called PRO measures (PROMs) completed di-
rectly by the patient without interpretation from a clinician or
anyone else.1 Collectively, PROMs can inform clinicians about
the patient’s status in a variety of dimensions, including how
they are tolerating their current treatment. Treatment tol-
erability, a term typically used in the context of drug evalu-
ation, refers to the degree to which symptomatic and
nonsymptomatic adverse events affect the ability or desire of
construct that includes clinician-reported measures of safety
and patient-reported measures of well-being.
As shown in Figure 1, PROs are one of several clinical out-
come measures available for research and clinical practice
for assessing patients’ well-being and tolerance of
treatments.3 In the context of clinical trials, PROs inform
how a group of people tolerate treatment, whereas in clinical
practice, PROs inform how individual people feel and
function, and how treatments change the way people feel
and function.
PROs can be used in several different ways to inform an
individual patient’s care. This can include screening for
problems, symptom monitoring and management, assess-
ing needs, promoting patient-centered care, assessing
outcomes (eg, the effectiveness of a treatment), or
informing shared decision making. In this chapter, we will
focus mainly on the use of PROs for symptom monitoring
and management.
WHY IS IT IMPORTANT TO COLLECT PROs IN ROUTINE
CARE? A CONSUMER PERSPECTIVE
My cancer has returned and I am back in clinic. This time, the
treatments are more aggressive. My doctor insists that he

asco.org/edbook | Volume 44, Issue 3 | 1

 Lai-Kwon et al
PRACTICAL APPLICATIONS
• We highlight the importance of collecting patient-
reported outcomes (PROs) from the patient’s
perspective.
• We provide a step-by-step guide to identifying clin-
ically relevant symptoms and side effects for PRO
symptom monitoring, assessing the relevance of
existing PRO measures to assess these issues, and
creating customized item lists from item libraries to
fill in the gaps.
• We highlight key pearls and pitfalls in the interpre-
tation of PRO data, including how data display
strategies can improve interpretation of PRO scores.
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
understands that my top priority, behind not dying, is my
quality of life. As always, the visit begins when I am handed a
tablet and asked to fill out PROMs. As I start checking boxes,
my husband looks around anxiously and scrolls through
social media on his phone. I know the drill…
• Fatigue? I used to be a triathlete and now am too exhausted to
walk up the stairs.
• Depression/anxiety? Who wouldn’t be in my situation?
• Problems sleeping? Most nights, I wake up multiple times,
crying and hiding from a world that thinks I am brave.
• Diarrhea? Yes! Taking my daughter to school was one of the
few activities that I could still do until this became a regular
issue. Now, I am worried about a blowout while I am stuck in
the carpool line.
• Dyspareunia? I cannot remember the last time I had any
interest in being touched.
• Fertility issues? Thanks for the reminder that treatment
robbed me of the ability to grow my family.
• Erectile disfunction? This is my favorite question! As a cis-
gender female being treated for breast cancer, I audibly laugh
at this question.
After 20 minutes of PROMs, I am brought back to the ex-
amination room where I wait for my oncologist. During my
10-minute visit, he reviews my labs and makes small talk
with my husband. Without looking at the PROMs I just
completed, or even looking away from the computer, he asks
me about my fatigue, nausea, and pain. He stands up to leave,
and on his way out, says, “you’re doing great.” I must look
confused because he continues, “This is a great clinical
response, keep up the great work.” See you in a month when
we can do this all over again. Doesn’t the doctor see how much I
am struggling? Doesn’t he care that the drugs he’s giving me are
keeping me from driving my daughter to school? Why do I have
to answer a question about erectile dysfunction? Is anybody even
going to see it?
2 | © 2024 by American Society of Clinical Oncology
How We Can Do Better
I am not just a patient, but also someone who has worked in
clinical and research oncology care for 20 years. I understand
the complexities of efficient clinical flow and validated
metrics, not to mention overstretched clinicians and reim-
bursement challenges. PROMs have great potential to im-
prove the quality of my care. However, patients need to feel
that their investment in completing PROMs is meaningful
and affects their care.
When I was a patient, these are things I wish my health care
professionals knew:
Ask the right questions—Tailor the questions to the patient.
Just as I was irritated about being asked about erectile
disfunction and saddened by the reminder that I can no
longer bare children, irrelevant or insensitive questions
show a lack of individualized care.
Ask at the right time—If you want a patient to thoughtfully
answer PROMs, invest clinical time to review the answers
and use them to tangibly guide care. Not only does this
convey to the patient the importance of his/her overall
wellness, but it also demonstrates that PROs are impor-
tant. Collaborate with your administration and infor-
mation technology (IT) teams to develop tools to collect
and review PROMs before a visit. As real as we know these
challenges to be, already overwhelmed and scared pa-
tients are rarely sympathetic to technology constraints,
electronic medical record (EMR) integration, and clinical
workflow challenges.
Ask my caregivers—As much as I believe that I was a model
patient (well-educated, unemotional, cheerful, and
compliant), my husband, mother, and best friend would
tell a different story. Ask them for that story. My husband
knew I was depressed and cried at night but let me pretend
that I was always brave; my mother saw I struggled with
cognitive disfunction as my executive functioning slip-
ped; and my best friend knew I spaced out my antiemetics
to avoid the side effects. Clinics should broaden PROMs to
solicit perspective from the caregiver—not just to gain
additional insight, but also to give a platform to their
voice.
Show results—Patients like to know how their symptoms
compare with previous visits so they can assess how
treatment is going. They also like to know how they
compare with similarly situated patients. If possible, show
patients individualized and group-level data so that they
can monitor their progress. This will also demonstrate
that clinicians look at PROs and consider quality of life
when formulating a treatment plan.
PROMs can be a useful tool for patients, especially if it allows
them to track symptoms, engage more effectively with
clinicians, and feel like an equal participant in their own care.
If we are asking patients to complete PROMs, it is our re-
sponsibility to ensure they are relevant, and neither cum-
bersome nor redundant. Clinicians must first demonstrate
their utility and the potential to improve care delivery. If we
 A Practical Guide to PROs in Routine Care
FIG 1. Types of clinical outcomes assessment.3
do not, patients will feel unheard and disrespected. They will
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
invest in PROMs as much as they think their clinicians do.
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
SELECTING PATIENT-REPORTED OUTCOMES FOR
SYMPTOM MONITORING IN ROUTINE CARE
To successfully meet the challenge of asking the right
questions about the right symptoms at the right time, this
section provides an overview of how to select PROs for
symptom monitoring in routine care.
What Is Electronic Patient-Reported Outcome
Symptom Monitoring?
Electronic patient-reported outcome (ePRO) symptom
monitoring typically involves administering electronic
surveys containing questions about symptoms, adminis-
tered via a computer, smartphone app, or automated tele-
phone system. People are asked to report symptoms on a
regular basis (between clinic visits and/or at clinic visits)
with reminders sent via e-mail, text, or automated tele-
phone system. Alerts are sent to clinicians for severe or
worsening symptoms.
Multiple studies have demonstrated that systematic moni-
toring of patients using PROs can improve symptom control,4-8
physical function,9-11 treatment adherence,8-13 patient
satisfaction,7,11,13 patient self-efficacy,6,10 health-related quality
of life,8-11 cost-effectiveness,14,15 and overall survival,16-19 and
reduce emergency department presentations8,9,13,18,20-22 and
hospital admissions.18,22 Although systems can be deployed in
people with both early-stage and advanced disease, those with
advanced disease may be more symptomatic from their cancer
and its treatment, and may therefore benefit more from
proactive symptom management.23-25
ePRO symptom monitoring is recommended by the Euro-
pean Society of Medical Oncology Clinical Practice Guide-
lines24 and as part of value-based health care initiatives in
the United States.26,27 In the United States, the commence-
ment of the Center for Medicare and Medicaid Innovation’s
ASCO Educational Book
(CMMI) new voluntary Enhancing Oncology Model has
created additional impetus for ePROs to be incorporated into
routine care.
Selection of Patient-Reported Outcomes for Symptom
Monitoring—A Practical Guide for Clinicians
A critical preliminary step in setting up ePRO symptom
monitoring is deciding what symptoms and side effects to
monitor for. This might include proximal effects such as
cancer symptoms, disease-specific symptoms, and/or
treatment-specific side effects. Proximal effects occur be-
cause of the direct effects of the cancer and/or its treatment,
but may also affect a person’s ability to function and their
overall well-being (ie, cause distal effects).28 The proximal/
distal distinction is important because distal outcomes will
be influenced by factors external to health care.28
The following steps are intended as a practical guide for
clinicians seeking to select PROs for symptom monitoring
with examples from the literature.
1. Engage Key Stakeholders
Key stakeholders who will be involved in collecting and
interpreting PROs should be involved at all stages of the PRO
selection process. Including a range of perspectives will
ensure that the selected PROs meet the needs of all relevant
stakeholders. Stakeholders may include patients, caregivers,
health care professionals (eg, oncologists, nurses, and
pharmacists), IT specialists, and administrators. A PRO
methodologist can also help support the selection of ap-
propriate PROMs and highlight existing resources to guide
this process.
The PROTEUS Consortium (Patient-Reported Outcomes
Tools: Engaging Users & Stakeholders) is an international
body of more than 50 patient, clinician, research, health
system, industry, policy, government, and regulatory groups
who help navigate the use of PROs. Its resources are de-
scribed further below. The PROTEUS-Practice guide29
asco.org/edbook | Volume 44, Issue 3 | 3
 Lai-Kwon et al
provides further information about how to identify and
engage stakeholders. Guidelines for promoting diversity,
equity, and inclusion should also be considered when
identifying and engaging with stakeholders.
2. Identify Clinically Relevant Symptoms and Side Effects
International guidelines30-35 provide a useful framework for
selecting appropriate symptoms and side effects for moni-
toring. This could include:
• Review of the literature, including clinical trials of the
treatment in question, relevant clinical practice guide-
lines, literature (including qualitative studies) reporting
on symptoms or side effects experienced by the population
in question, and product information.
• Review of medical records. This involves reviewing medical
records of the relevant population and extraction of
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
clinically important symptoms and side effects.
• Review of existing PROMs and core outcome sets for the
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
relevant population and extraction of items from these
measures/sets. Examples of core outcome sets include
those developed by the International Consortium for
Health Outcomes Measurement (ICHOM) or the National
Cancer Institute’s core set of 12 symptoms for measure-
ment in all patients with adult cancer.36
• Interviews with key stakeholders to elicit further issues. Al-
though this may not always be feasible when designing
systems for use in routine care, this is particularly relevant
if there has been minimal patient representation in the
previous steps.
Examples. Tolstrup et al37 sought to identify relevant
symptoms for monitoring in people with melanoma re-
ceiving immunotherapy. The authors conducted a literature
review, a medical record audit of people receiving immu-
notherapy at their institution to assess whether this aligned
with the symptoms reported in the literature, and a review of
immunotherapy product information from the European
Medicines Agency.
Taarnhoj et al sought to identify relevant symptoms for
monitoring in people with bladder cancer receiving che-
motherapy or immunotherapy.38 The authors conducted a
literature review of toxicity reporting from phase II and III
trials of immunotherapy in people with bladder cancer, a
medical record audit, a review of product information from
the European Medicines Agency and the US Food and Drug
Administration for chemotherapies, and interviews with
patients receiving chemotherapies.
3. Refine the List of Clinically Relevant Symptoms and Side
Effects
Once a list of relevant PROs has been compiled from the
various sources outlined in step 2, this will often be extensive
and may require reduction to balance adequate coverage of
clinically relevant symptoms and side effects against
4 | © 2024 by American Society of Clinical Oncology
minimizing patient and administrative burden and ensuring
adequate completion.37,39 This could be done via either:
• A modified Delphi survey with key stakeholders to identify
and prioritize the most clinically relevant symptoms and
side effects to monitor.
• Convening a panel of experts (patients, caregivers, and
health care professionals) to select the most clinically
relevant symptoms and side effects to monitor.
Examples. Da Silva Lopes et al39 conducted a modified
Delphi study to determine which PRO-CTCAE symptom
terms were relevant for people receiving immune checkpoint
inhibitors. The study included physicians, nurses, and pa-
tient advocates. This process identified 30 PRO-CTCAE terms
that were important to include, as well as 45 additional side
effects not covered by the PRO-CTCAE that could be de-
veloped in the future for people receiving immune check-
point inhibitors.
Msaouel et al12 used a team of clinical experts to identify
common and/or severe toxicities and their corresponding
clinical symptoms associated with immune checkpoint
inhibitors.
4. Assess the Appropriateness of Existing PROMs for
Monitoring Selected Symptoms and Side Effects
The PROTEUS Practice Guide29 builds on the International
Society for Quality of Life Research (ISOQOL) guidelines40,41
to provide key criteria to consider when selecting from
existing PROMs.
Critical considerations include understanding why you are
collecting PROs, what information you feel would be most
clinically relevant, and how you plan to use the PRO data.
The ideal PROM will address the reason why you are col-
lecting PROs. For example, although you may want to
primarily collect ePROs for symptom monitoring at the
individual level, you may also wish to use the data for
research/quality assurance purposes. This may influence
which PROM(s) you select. PROMs must also be psycho-
metrically robust and feasibly implemented in the context
of interest. However, there is often no single PROM that will
assess all the identified symptoms and side effects.
Therefore, you may need to use a combination of existing
PROMs and/or customized item lists (as described in
step 5).
Where can I find existing PROMs? The Patient-
Reported Outcome and Quality of Life Instruments Data-
base (PROQOLID)42 is a comprehensive online database
where you can search for existing PROMs. The free-access
level provides basic information such as name of authors/
developers, copyright notice, an overview of the PROM’s
content, therapeutic area, therapeutic indication, language
of development, and references. Common PROMs include
Patient-Reported Outcomes Measurement Information
 A Practical Guide to PROs in Routine Care
System (PROMIS) measures, Functional Assessment of
Chronic Illness Therapy (FACIT) measures, and the Euro-
pean Organisation for Research and Treatment of Cancer
(EORTC) measures. Item libraries are also discussed in step 5.
What should I consider when choosing a PROM?
(a) Is the PROM’s content relevant?
• On the face of it, does it assess the symptoms and side
effects relevant to my intended application (eg, disease
population, treatment(s))?
• Does it evaluate the relevant aspects of these symp-
toms and side effects, eg, frequency, severity, and
interference with activities of daily living?
(b) Does the PROM have evidence for its psychometric
properties in your specific patient group/clinical context?
• Is there published evidence about its reliability, val-
idity, and clinical utility (eg, is it able to discriminate
between clinical groups known to differ, is it re-
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
sponsive to clinically significant changes as well as
changes over time) within the same (or similar) pa-
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
tient population and clinical context?
• Is the recall period appropriate for what you are trying
to measure?
• Is it at an appropriate reading level for your patient
group?
• Does it provide normative values to help you interpret
your results (eg, thresholds for clinical significance,
normative data)?
(c) Is it feasible to implement?
• How burdensome is it for patients to complete? Pa-
tients are often willing to complete longer PROMs if
they understand why they are completing them and
the questions are relevant and meaningful to them.
• Cost/licensing—some PROMs may require permis-
sions to use or have fees associated with their use.
• Availability of translations—many PROMs are
available in multiple languages. If your desired lan-
guage version is not available, international guide-
lines are available for cross-cultural and linguistic
translation.43
• Compatibility with your ePRO system
Example. Cancer Care Ontario (CCO) used the Edmonton
Symptom Assessment Scale (ESAS), a PROM assessing ge-
neric cancer-related symptoms in all ambulatory clinics.
Recognizing the need for specific PROMs in selected sites
(eg, head and neck cancers),23 they developed a streamlined
approach to the selection and implementation of disease-
specific PROMs, creating a PROM advisory committee with
methodological expertise, as well as regional and patient
representation. Disease sites were selected on the basis of
disease burden, symptom burden, and the window of op-
portunity to ease implementation. A literature review was
conducted to identify candidate PROMs. PROMs were se-
lected on the basis of symptom coverage, usability (the
number of questions), and psychometric properties. The
chosen PROMs were then piloted to evaluate their accept-
ability to patients and clinicians, impact on outcomes, and to
identify sustainability issues. This process was initially
ASCO Educational Book
applied in the setting of prostate cancer, where symptoms
not covered by the ESAS were identified (eg, urinary diffi-
culties, bowel symptoms, and sexual health issues). The
Expanded Prostate Cancer Index Clinical (EPIC-16) was then
selected and implemented province-wide.
5. Create Customized Item Lists From Item Libraries to
Cover Any Missing Symptoms or Side Effects
Often, not all relevant symptoms and side effects will be
covered by an existing PROM. This is particularly pertinent
when assessing treatment-related side effects of novel
therapies. A solution to this is creating customized item lists
from an item library.
Item libraries contain items drawn from existing static
PROMs, such as the EORTC Item Library or FACIT searchable
library, or developed as a standalone system, such as the
Patient-Reported Outcome version of the Common Termi-
nology Criteria for Adverse Events (PRO-CTCAE). Items can
be flexibly combined to create customized item lists. The
benefits of using items from an item library, rather than
creating your own items, is that each item has been rigor-
ously developed to ensure its content validity and is often
translated in a range of languages.
Examples of item libraries.
• EORTC item library—The EORTC Item Library is an online
interactive catalog of 950 items from the EORTC Quality of
Life Group’s PROMs.44,45 It allows more flexible use of the
static questionnaires by enabling users to create cus-
tomized item lists.46 Access to the item library can be
requested for free for academic use (https://
itemlibrary.eortc.org/)
• FACIT searchable library—The FACIT searchable library
comprises over 700 items from FACIT’s PROMs, enabling
the creation of customized lists.47 Access is free for aca-
demic use (https://wizard.facit.org/)
• PRO-CTCAE—The PRO-CTCAE is a library of 124 items
measuring 78 symptomatic adverse events available from
the US National Cancer Institute.48-50 For each adverse
event, up to three items are administered to evaluate the
side effect’s frequency, severity, and interference with
activities of daily living. Access is free for academic use
(https://healthcaredelivery.cancer.gov/pro-ctcae/
instruments/pro-ctcae/pro-ctcae_english.pdf)
Example. Møller et al51 developed an item set for acute
treatment toxicities of pelvic online magnetic resonance–
guided radiotherapy. Eighteen issues were selected using a
literature review and chart audit. The authors then reviewed
the PRO-CTCAE and selected relevant items. However, some
issues were not included in the PRO-CTCAE. These issues
were then drawn from the EORTC item library to ensure all
relevant symptoms were included using plain language,
validated questions. Where symptoms were available in both
item libraries, the face validity and wording of the item
influenced the item selected.
asco.org/edbook | Volume 44, Issue 3 | 5
 Lai-Kwon et al
Conclusion
We have provided a step-by-step guide to identifying
clinically relevant symptoms and side effects for ePRO
symptom monitoring, assessing the relevance of existing
PROMs to assess these issues, and creating customized item
lists to fill the gaps. Key stakeholders should be involved
throughout this process to ensure the final PROM(s) and/or
item lists are relevant and acceptable to all stakeholders. This
process highlights the importance of rigorously identifying
clinically important symptoms and side effects first before
selecting PROMs, rather than simply choosing an existing
PROM off the shelf. This will ensure the final list of PROs for
monitoring is both content-valid and has clinical utility.
PEARLS AND PITFALLS IN INTERPRETING PRO SCORES
Realizing the benefits of PROs requires careful attention to
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
using best practices for their implementation and use. This
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
section highlights several key pearls and pitfalls in the
implementation and interpretation of patients’ PRO data.
Key Pearls
1. Collecting PROs is feasible and helpful for improving
patients’ quality of life and the quality of care in clinical
practice.
The previous sections outline the benefits of using PROs in
practice, and provide examples of PRO system feasibility,
emphasizing that their advantages greatly outweigh their
challenges. Although many clinicians feel their clinical skills
are sufficient to screen and monitor patients’ well-being,
multiple studies show that clinicians often miss symptoms
that are important to patients and that impact on their well-
being.52-57 Moreover, symptom monitoring between clinic
visits using PROs can help clinicians detect symptoms and
intervene earlier. It can also reduce follow-up visits for
patients who report feeling well.
2. Clinical interpretation of PRO scores can be challenging. As
with other laboratory tests that clinicians learn to use,
interpretation requires familiarization and engagement in
learning about PROs.
There are some potentially confusing aspects of interpreting
PRO scores. For some PROMs, low scores indicate good
symptom control (eg, pain on a scale from 0 to 10), whereas
high scores indicate good patient functioning. Some PROMs
are scored using a range from 0 to 100, whereas others are
statistically normed such that a score of 50 represents the
population average score, and each patients’ score is re-
ported as a t-score (like reports of osteoporosis severity
compared with age-group averages on a bone density scan).
Fortunately, training, repetition, familiarization, and ap-
propriate PRO data display strategies will overcome most of
these challenges. Studies show that data display strategies
can aid interpretation and improve understanding of pa-
tients’ scores.58 Strategies include clear labeling of the
6 | © 2024 by American Society of Clinical Oncology
domain being displayed; visual cues on the score axis to
indicate the scale breadth and directionality; use of cutpoints
or color coding to indicate normal score ranges (as well as
intermediate score ranges where data are available to sup-
port their placement); use of alert thresholds to indicate
potentially concerning score values; and longitudinal display
of data to create a context for interpretation of current
results.29,41,58,59 Integration of PRO results into the EMR to
enable review of PROs alongside laboratory and imaging
results can also promote use.29,60-63 However, effective in-
formation processes can be used employing paper records
as well.
Figure 2 shows an example of a display of a patient’s
symptom scores over time. In this PRO system from Alberta
Heath Services in Canada,64 nine symptom scores obtained
with the ESAS are plotted over time. Each score is annotated
on the line for each symptom (instead of a y-axis label), and
color coding is used to indicate directionality as well as
potentially concerning scores. This display forms part of a
larger dashboard of clinical results.
Figure 3 shows a different PRO result display within the
OncoPRO system.65 This clinician-facing dashboard shows
13 PRO-CTCAE symptoms, three of which are reported by the
patient as being present. Longitudinal results are plotted
over time, with severity/frequency results plotted on the left
y-axis and interference scores plotted on the right y-axis.
For example, on the day of the visit, the patient reports very
severe numbness and tingling with quite a bit of interference.
3. Optimal benefits from PROs require best practices, effort,
and stakeholder engagement
Although many health care systems are enthusiastic about
using PROs to inform care, many also lack an understanding
of the considerations involved in designing, implementing,
and managing PRO systems. Moreover, what constitutes an
ideal PRO system varies depending on the practice setting
and the patient population it serves.
To inform this process, the PROTEUS Consortium developed
a framework of key considerations for clinical PRO systems
(Fig 4).66 The Framework builds on the topics covered across
five foundational resources41,58,59,61,67 and presents key
considerations for designing, implementing, and managing
PRO systems. These include designing the goals of the
system which will drive issues of feasibility, the target
population, the selection of meaningful PROMs, and the
optimal timing and frequency of administration and clinical
review.
The PROTEUS Guide to Implementing Patient-Reported
Outcomes in Clinical Practice: A Synthesis of Resources
(The PROTEUS-Practice Guide) is a free resource that
operationalizes the above framework by providing a practical
synthesis of previous published guidance.29 The guide
comprises three sections addressing considerations of the
 A Practical Guide to PROs in Routine Care
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.

FIG 2. An example of PRO data display—Alberta Health Services Longitudinal Symptom Scores. ESAS,
Edmonton Symptom Assessment Scale; PRO, patient-reported outcome.

framework, providing a range of options where no single
approach is necessarily optimal.
Finally, OncoPRO is a publicly funded national program in
the United States supported by the Patient-Centered Out-
comes Research Institute and administered by the University
of North Carolina and the University of Alabama at Bir-
mingham, in partnership with ASCO, the American Cancer
Society (ACS), and major EMR and ePRO software vendors.65
OncoPRO aims to help practices with all aspects of designing,
onboarding, monitoring, and sustaining ePROs in their
clinical oncology practices for monitoring symptoms (and

ASCO Educational Book asco.org/edbook | Volume 44, Issue 3 | 7

 Lai-Kwon et al
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.

FIG 3. An example of PRO data display—OncoPRO.65 PRO, patient-reported outcome.

other key areas such as health-related social needs), in-
cluding the provision of guidance of ePRO implantation
experts and facilitating coordination with EMR and ePRO
vendors.
Key Pitfalls
Reports of PRO implementation have consistently demon-
strated several potential areas that can threaten the creation
of an effective PRO system in clinical practice. These pitfalls
include not just those relevant to PRO score interpretation
but also those relevant to the context in which the scores are
used.
1. The selected PROM may be off the shelf and not fit for
purpose.
As noted by patients, a universal PROM designed for all
patients with cancer may not be as useful as one enhanced
with items specific to the patient population and/or treat-
ment of interest. This trade-off reflects the need to balance

FIG 4. Key considerations for integrating PROs into clinical systems.66 EHR, electronic health record; PROs,
patient-reported outcomes.

8 | © 2024 by American Society of Clinical Oncology

 A Practical Guide to PROs in Routine Care
the benefits of a single PROM approach across patients
against the complexities of clinic-specific PROMs.
2. The selected PROM and its scores may not be understood
by either the patient or the physician.
For example, some patients find normed scores confusing,
wondering why they only scored 60 points (a full standard
deviation above the norm) when they feel quite well. As noted
earlier, clinicians are familiar with normed laboratory results
accompanied by a clinical interpretation (eg, a bone density
t-score interpreted as mild osteoporosis for age). Regardless
of the scale used for the PRO, education, effective and
consistent data display strategies, and summative clinical
interpretations can improve understanding of the scores.
3. The scores are not acknowledged in the eyes of the patient
or their caregivers.
As highlighted earlier, a clinician’s reference to today’s scores
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145
in the clinical encounter reinforces the importance of filling
out PROs to patients. Conversely, patients may lose interest in
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.
PRO reporting if they do not perceive the report to be useful to
their clinical team. Referring to PRO results during the clinic
visit will reinforce patient understanding of their utility
4. Many clinicians initially see little value-add from a PRO
system.
This perception may be reinforced if the implementation
process is flawed. Conversely, many clinicians testify to the
added value of PROs in making their clinical practice of
higher quality and more efficient when PROs are imple-
mented appropriately.
5. Clinical alerts of high scores are both potentially helpful
and potentially problematic.
This phenomenon can lead to alert fatigue or, conversely, a
lack of confidence in the sensitivity of alerts. Stakeholder
engagement, pilot evaluations, and iterative adjustments are
usually required to find an appropriate alert threshold
strategy that optimally balances too few (false negative) with
too many (false positive) alerts.
6. The clinical care path relevant to a concerning PRO score
may be unclear.
AFFILIATIONS
1
Department of Medical Oncology, Peter MacCallum Cancer Centre,
Melbourne, Australia
2
Department of Health Services Research, Peter MacCallum Cancer
Centre, Melbourne, Australia
3
PROTEUS Consortium
4
Sydney Quality of Life Office, Susan Wakil School of Nursing and
Midwifery, Faculty of Medicine and Health, The University of Sydney,
Sydney, Australia
5 Relationships are self-held unless noted. I 5 Immediate Family
The Ohio State University College of Medicine, Columbus, OH
6 Member, Inst 5 My Institution. Relationships may not relate to the
Queen’s Cancer Research Institute, Queen’s University, Kingston,
Ontario, Canada
ASCO Educational Book
Studies indicate that health care professionals sometimes
lack confidence in how to act on abnormal PRO scores. The
creation of care pathways and treatment guidelines can
mitigate these concerns. Some patients have chronic con-
ditions where scores are consistently abnormal. The effective
use of longitudinal data display and clinical annotation can
be useful in these circumstances.
7. The PRO scores are not integrated with clinical workflow.
Deficits in implementation can lead to PRO scores being
poorly integrated with busy clinic workflows. PROs inte-
grated into the EMR need to be entered such that the current
scores are available to the clinician reviewing data for that
encounter, or conversely, are available in paper form to
complement the review of the EMR. Since many evaluations
of ePRO systems show that only some patients will complete
their PROMs online, hybrid data collection systems may be
necessary to accommodate all patients.
8. Error happens!
Some patients may make errors in completing the PROM (eg,
unwittingly invert the direction of the response scale) or may
decline cognitively, thus limiting their ability to complete
the PROM accurately. As with any laboratory result, an
unexpected PRO result requires clinical assessment in the
context of the patient’s health status.
CONCLUSION
Symptom monitoring using PROs in people with advanced
cancer is highly desired and valued by patients. We have
provided a practical guide to identifying clinically relevant
symptoms and side effects for symptom monitoring in
conjunction with key stakeholders, assessing the relevance
of existing PROMs to assess these issues, and creating item
lists to fill in the gaps. We have also highlighted key pearls
and pitfalls to interpreting and implementing PROs in
routine care, including where existing guidance and re-
sources are available. Despite its challenges, implementing
PROs in routine care remains a worthwhile investment in
optimal patient care.
CORRESPONDING AUTHOR
Julia Lai-Kwon, MBBS, BMedSci, MPH; Twitter: @julialaikwon; e-mail:
Julia.Lai-Kwon@petermac.org.
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS
OF INTEREST AND DATA AVAILABILITY STATEMENT
The following represents disclosure information provided by authors of
this manuscript. All relationships are considered compensated.
subject matter of this manuscript. For more information about ASCO’s
conflict of interest policy, please refer to www.asco.org/rwc.
asco.org/edbook | Volume 44, Issue 3 | 9
 Lai-Kwon et al

Claudia Rutherford Michael Brundage

Research Funding: Little Green Pharma Ltd Research Funding: Pfizer (Inst)

Norah Crossnohere No other potential conflicts of interest were reported.

Consulting or Advisory Role: Alcon (I), Boehringer Ingelheim, Pfizer
Travel, Accommodations, Expenses: Alcon

REFERENCES
1. US Food and Drug Administration: Guidance for Industry—Patient-Reported Outcomes Measures: Use in Medical Product Development to Support Labeling Claims. Silver Spring, MD, 2009. https://
www.fda.gov/media/77832/download
2. Basch E, Campbell A, Hudgens S, et al: Broadening the definition of tolerability in cancer clinical trials to better measure the patient experience. Friends of Cancer Research, 2018. https://
friendsofcancerresearch.org/wp-content/uploads/Comparative-Tolerability-Whitepaper_FINAL.pdf
3. Center for Drug Evaluation and Research: Clinical Outcome Assessment Compendium. Food and Drug Administration. 2021. [Available from: https://www.fda.gov/drugs/development-resources/
clinical-outcome-assessment-compendium
4. Strasser F, Blum D, von Moos R, et al: The effect of real-time electronic monitoring of patient-reported symptoms and clinical syndromes in outpatient workflow of medical oncologists: E-MOSAIC,
a multicenter cluster-randomized phase III study (SAKK 95/06). Ann Oncol 27:324-332, 2016
5. Diplock BD, McGarragle KMC, Mueller WA, et al: The impact of automated screening with Edmonton Symptom Assessment System (ESAS) on health-related quality of life, supportive care needs,
and patient satisfaction with care in 268 ambulatory cancer patients. Support Care Cancer 27:209-218, 2019
6. Maguire R, McCann L, Kotronoulas G, et al: Real time remote symptom monitoring during chemotherapy for cancer: European multicentre randomised controlled trial (eSMART). BMJ 374:n1647,
2021
7. Rasschaert M, Vulsteke C, De Keersmaeker S, et al: AMTRA: A multicentered experience of a web-based monitoring and tailored toxicity management system for cancer patients. Support Care
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145

Cancer 29:859-867, 2021

8. Zhang L, Zhang X, Shen L, et al: Efficiency of electronic health record assessment of patient-reported outcomes after cancer immunotherapy: A randomized clinical trial. JAMA Netw Open 5:
e224427-e, 2022
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.

9. Basch E, Deal AM, Kris MG, et al: Symptom monitoring with patient-reported outcomes during routine cancer treatment: A randomized controlled trial. J Clin Oncol 34:557-565, 2016
10. Absolom K, Warrington L, Hudson E, et al: Phase III randomized controlled trial of eRAPID: eHealth intervention during chemotherapy. J Clin Oncol 39:734-747, 2021
11. Basch E, Schrag D, Jansen J, et al: Digital symptom monitoring with patient-reported outcomes in community oncology practices: A U.S. National cluster randomized trial. J Clin Oncol 39, 2021
(suppl 36; abstr 349527)
12. Msaouel P, Oromendia C, Siefker-Radtke AO, et al: Evaluation of technology-enabled monitoring of patient-reported outcomes to detect and treat toxic effects linked to immune checkpoint
inhibitors. JAMA Netw Open 4:e2122998, 2021
13. Mir O, Ferrua M, Fourcade A, et al: Digital remote monitoring plus usual care versus usual care in patients treated with oral anticancer agents: The randomized phase 3 CAPRI trial. Nat Med, 2022
14. Nixon NA, Spackman E, Clement F, et al: Cost-effectiveness of symptom monitoring with patient-reported outcomes during routine cancer treatment. J Cancer Policy 15:32-36, 2018
15. Lizée T, Basch E, Trémolières P, et al: Cost-effectiveness of web-based patient-reported outcome surveillance in patients with lung cancer. J Thorac Oncol 14:1012-1020, 2019
16. Basch E, Deal AM, Dueck AC, et al: Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318:197-198, 2017
17. Denis F, Lethrosne C, Pourel N, et al: Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. J Natl Cancer Inst 109:djx029, 2017
18. Barbera L, Sutradhar R, Seow H, et al: Impact of Standardized Edmonton Symptom Assessment System use on emergency department visits and hospitalization: Results of a population-based
retrospective matched cohort analysis. JCO Oncol Pract 16:e958-e65, 2020
19. Caminiti C, Maglietta G, Diodati F, et al: The effects of patient-reported outcome screening on the survival of people with cancer: A systematic review and meta-analysis. Cancers 14:5470, 2022
20. Girgis A, Durcinoska I, Arnold A, et al: Web-based Patient-Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care): Multicenter pragmatic nonrandomized trial. J Med
Internet Res 22:e19685, 2020
21. Howell D, Li M, Sutradhar R, et al: Integration of patient-reported outcomes (PROs) for personalized symptom management in "real-world" oncology practices: A population-based cohort
comparison study of impact on healthcare utilization. Support Care Cancer 28:4933-4942, 2020
22. Kolodziej MA, Kwiatkowsky L, Parrinello C, et al: ePRO-based digital symptom monitoring in a community oncology practice to reduce emergency room and inpatient utilization. J Clin Oncol 40,
2022 (suppl 16; abstr 1508)
23. Basch E, Barbera L, Kerrigan CL, et al: Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 38:122-134, 2018
24. Di Maio M, Basch E, Denis F, et al: The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO clinical practice guideline. Ann Oncol 33:878-892, 2022
25. Caston NE, Franks JA, Balas N, et al: Evaluating nurses’ time to response by severity and cancer stage in a remote symptom monitoring program for patients with breast cancer. JCO Clin Cancer
Inform 10.1200/CCI.23.00015
26. Woofter K, Kennedy EB, Adelson K, et al: Oncology medical home: ASCO and COA standards. JCO Oncol Pract 17:475-492, 2021
27. Basch E, Wilfong L, Schrag D: Adding patient-reported outcomes to Medicare’s oncology value-based payment model. JAMA 323:213-214, 2020
28. Brenner MH, Curbow B, Legro MW: The proximal-distal continuum of multiple health outcome measures: The case of cataract surgery. Med Care 33:AS236-AS244, 1995 (4 suppl)
29. Crossnohere N, Brundage M, Snyder C, et al: The PROTEUS Guide to Implementing Patient-Reported Outcomes in Clinical Practice: A Synthesis of Resources, 2023.
www.TheProteusConsortium.org
30. Johnson C, Aaronson N, Blazeby JM, et al: EORTC Quality of Life Group Guidelines for Developing Questionnaire Modules, 2011. https://www.eortc.org/app/uploads/sites/2/2018/02/
guidelines_for_developing_questionnaire-_final.pdf
31. Group EQoL: EORTC Quality of Life Group Module Development Guidelines. Brussels, Belgium, 2021
32. Webster K, Cella D, Yost K: The Functional Assessment of Chronic Illness Therapy (FACIT) Measurement System: Properties, applications, and interpretation. Health Qual Life Outcomes 1:79, 2003
33. Coons SJ, Gwaltney CJ, Hays RD, et al: Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO)
measures: ISPOR ePRO Good Research Practices Task Force report. Value health 12:419-429, 2009
34. Wild D, Grove A, Martin M, et al: Principles of Good Practice for the Translation and Cultural Adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for
Translation and Cultural Adaptation. Value health 8:94-104, 2005
35. Rothman M, Burke L, Erickson P, et al: Use of existing patient-reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluating and documenting
content validity for the use of existing instruments and their modification PRO task force report. Value health 12:1075-1083, 2009
36. Reeve BB, Mitchell SA, Dueck AC, et al: Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials. J Natl Cancer Inst 106:dju129, 2014
37. Tolstrup LK, Bastholt L, Zwisler A-D, et al: Selection of patient reported outcomes questions reflecting symptoms for patients with metastatic melanoma receiving immunotherapy. J Patient Rep
Outcomes 3:19, 2019
38. Taarnhøj GA, Johansen C, Lindberg H, et al: Patient reported symptoms associated with quality of life during chemo- or immunotherapy for bladder cancer patients with advanced disease. Cancer
Med 9:3078-3087, 2020
39. Da Silva Lopes AM, Colomer-Lahiguera S, Mederos Alfonso N, et al: Patient-reported outcomes for monitoring symptomatic toxicities in cancer patients treated with immune-checkpoint inhibitors:
A Delphi study. Eur J Cancer 157:225-237, 2021
40. Chan EKH, Edwards TC, Haywood K, et al: Implementing patient-reported outcome measures in clinical practice: A companion guide to the ISOQOL user’s guide. Qual Life Res 28:621-627, 2019
41. Aaronson N, Elliot T, Greenhalgh J, et al: User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice. International Society for Quality of Life Research, 2015. https://
www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf
42. Trust MR: Patient-Reported Outcome and Quality of Life Instrument Database (PROQOLID). Lyon, France, 2008. https://www.qolid.org/
43. Kulis D, Bottomley A, Velikova G, et al: EORTC Quality of Life Group Translation Procedure, 2017. https://www.eortc.org/app/uploads/sites/2/2018/02/translation_manual_2017.pdf
44. EORTC: EORTC Quality of LIfe Group Item Library, 2018. http://www.eortc.be/itemlibrary/
45. Kulis D, Bottomley A, Whittaker C, et al: The use of the Eortc item library to supplement Eortc quality of life instruments. Value Health 20:A775, 2017

10 | © 2024 by American Society of Clinical Oncology

 A Practical Guide to PROs in Routine Care

46. Gilbert A, Piccinin C, Velikova G, et al: Linking the European Organisation for research and treatment of cancer item library to the common Terminology criteria for adverse events. J Clin Oncol 40:
3770-3780, 2022
47. FACIT Group: The FACIT Searchable Library, 2020. https://www.facit.org/facit-searchable-library
48. Dueck AC, Mendoza TR, Mitchell SA, et al: Validity and reliability of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events
(PRO-CTCAE) Patient-Reported Outcomes version of the CTCAE. JAMA Oncol 1:1051-1059, 2015
49. Basch E, Reeve BB, Mitchell SA, et al: Development of the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).
J Natl Cancer Inst 106:dju244, 2014
50. Hay JL, Atkinson TM, Reeve BB, et al: Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events
(PRO-CTCAE). Qual Life Res 23:257-269, 2014
51. Møller PK, Pappot H, Bernchou U, et al: Development of patient-reported outcomes item set to evaluate acute treatment toxicity to pelvic online magnetic resonance-guided radiotherapy. J Patient
Rep Outcomes 5:47, 2021
52. Atkinson TM, Li Y, Coffey CW, et al: Reliability of adverse symptom event reporting by clinicians. Qual Life Res 21:1159-1164, 2012
53. Basch E, Jia X, Heller G, et al: Adverse symptom event reporting by patients vs clinicians: Relationships with clinical outcomes. J Natl Cancer Inst 101:1624-1632, 2009
54. Fromme EK, Eilers KM, Mori M, et al: How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire
C30. J Clin Oncol 22:3485-3490, 2004
55. Basch E, Iasonos A, McDonough T, et al: Patient versus clinician symptom reporting using the national cancer Institute Common Terminology Criteria for Adverse Events: Results of a
questionnaire-based study. Lancet Oncol 7:903-909, 2006
56. Di Maio M, Gallo C, Leighl NB, et al: Symptomatic toxicities experienced during anticancer treatment: Agreement between patient and physician reporting in three randomized trials. J Clin Oncol 33:
910-915, 2015
57. Fares CM, Williamson TJ, Theisen MK, et al: Low concordance of patient-reported outcomes with clinical and clinical trial documentation. JCO Clin Cancer Inform 10.1200/CCI.18.00059
58. Snyder C, Smith K, Holzner B, et al: Making a picture worth a thousand numbers: Recommendations for graphically displaying patient-reported outcomes data. Qual Life Res 28:345-356, 2019
59. Snyder C, Brundage M, Rivera YM, et al: A PRO-cision medicine methods toolkit to address the challenges of personalizing cancer care using patient-reported outcomes: Introduction to the
supplement. Med Care 57(suppl 5 suppl 1):S1-S7, 2019
60. Fritz F, Dugas M: Are physicians interested in the quality of life of their patients? Usage of EHR-integrated patient reported outcomes data. Stud Health Technol Inform 192:1039, 2013
61. Snyder C, Wu AW: Users’ Guide to Integrating Patient-Reported Outcomes in Electronic Health Records. Baltimore, MD, Johns Hopkins University, 2017. http://www.pcori.org/document/users-
Downloaded from ascopubs.org by 181.91.85.145 on June 19, 2024 from 181.091.085.145

guide-integrating-patient-reported-outcomes-electronic-health-records
62. Gensheimer SG, Wu AW, Snyder CF, et al: Oh, the places we’ll go: Patient-reported outcomes and electronic health records. Patient 11:591-598, 2018
Copyright © 2024 American Society of Clinical Oncology. All rights reserved.

63. Zhang R, Burgess ER, Reddy MC, et al: Provider perspectives on the integration of patient-reported outcomes in an electronic health record. JAMIA open 2:73-80, 2019
64. Watson L, Delure A, Qi S, et al: Utilizing Patient Reported Outcome Measures (PROMs) in ambulatory oncology in Alberta: Digital reporting at the micro, meso and macro level. J Patient Rep
Outcomes 5:97, 2021
65. Institute P-COR: Implementation of Symptom Monitoring with Electronic Patient-Reported Outcomes (ePROs) During Cancer Treatment: The OncoPRO Initiative, 2023. www.pcori.org/research-
results/2023/implementation-symptom-monitoring-electronic-patient-reported-outcomes-epros-during-cancer-treatment-oncopro-initiative
66. Crossnohere N, Anderson N, Baumhauer J, et al: A framework for implementing patient-reported outcomes in clinical care: The PROTEUS-practice guide. Nat Med 10.1038/s41591-024-02909-8
[epub ahead of print on April 16, 2024]
67. LeRouge C, Austin E, Lee J, et al: ePROs in Clinical Care: Guidelines and Tools for Health Systems. Seattle, WA, University of Washington, 2020. epros.becertain.org

ASCO Educational Book asco.org/edbook | Volume 44, Issue 3 | 11