Received: 29 October 2020 Revised: 4 January 2021
DOI: 10.1002/alr.22782
ORIGINAL ARTICLE
Responsiveness and convergent validity of the chronic
rhinosinusitis patient-reported outcome (CRS-PRO)
measure in CRS patients undergoing endoscopic sinus
surgery
Katherine A. Lin BS1
Saied Ghadersohi MD1
David B. Conley MD1
Kevin C. Welch MD1
1Department of Otolaryngology–Head
and Neck Surgery, Northwestern
University, Feinberg School of Medicine,
Chicago, Illinois
2
Department of Medical Social Sciences,
Northwestern University, Feinberg School
of Medicine, Chicago, Illinois
3 Institute for Public Health and Medicine
(IPHAM)–Center for Patient-Centered
Outcomes, Northwestern University,
Feinberg School of Medicine, Chicago,
Illinois
Correspondence
Bruce K. Tan, MD, MS, Department of
Otolaryngology–Head and Neck Surgery,
Feinberg School of Medicine, Northwest-
ern University, 675 N St Clair St, Suite
15-200, Chicago, IL, 60611.
E-mail: b-tan@northwestern.edu
View this article online at wileyonlineli-
brary.com.
Funding information
National Institutes of Health,
Grant/Award Numbers: P01 AI145818-
Chronic Rhinosinusitis Integrative St, R01
AI134952, R01DC016645; Ernest S. Bazley
Foundation
1308 © 2021 ARS-AAOA, LLC
Accepted: 26 January 2021
Caroline P.E. Price BA1 Julia H. Huang DDS, MS1
David Cella PhD2,3 Robert C. Kern MD1
Stephanie Shintani-Smith MD, MS1,3
Bruce K. Tan MD, MS1
Background: The chronic rhinosinusitis patient-reported outcome (CRS-PRO)
measure is a 12-item measure with previously demonstrated validity in chronic
rhinosinusitis (CRS) patients receiving medical therapy. This study establishes
the factor structure, responsiveness, and convergent validity of the CRS-PRO fol-
lowing endoscopic sinus surgery (ESS).
Methods: Northwestern CRS Subject Registry patients had pre-ESS, 3-month
(n = 111; CRS without nasal polyps [CRSsNP] = 60, CRS with nasal polyps
[CRSwNP] = 51), and 6-month (n = 86; CRSsNP = 47, CRSwNP = 39) post-
ESS assessments where patients completed the CRS-PRO, 22-item Sino-Nasal
Outcome Test (SNOT-22), and four Patient-Reported Outcomes Measurement
(PROM) Information System (PROMIS) short forms (general health measures).
Patients had pre-ESS objective testing (endoscopic and radiographic assessment).
Factor analysis was conducted using principal axis factoring with varimax rota-
tion on the baseline CRS-PRO. The clinically important difference (CID) was esti-
mated using both distribution-based and anchor-based methods.
Results: Factor analysis found the CRS-PRO comprised the “rhino-
psychologic,” “facial discomfort,” and “cough” factors, which were responsive
to ESS and correlated with the other PROMs. The changes observed in the
CRS-PRO at 3 months had strong correlation with the corresponding changes
in SNOT-22 (r = 0.792, p < 0.0001) and moderate correlations with changes
in PROMIS fatigue and sleep domains. These changes had a very large effect
size (Cohen’s d 1.44) comparable to the longer SNOT-22 (Cohen’s d 1.41) with
slightly larger effect sizes observed in CRSwNP compared to CRSsNP patients.
Similar convergent validity and responsiveness were observed in the 6-month
data. The CRS-PRO CID was estimated to be between 5.0 and 7.5 (midpoint 6.0)
using distribution-based and anchor-based methods.
wileyonlinelibrary.com/journal/alr Int Forum Allergy Rhinol. 2021;11:1308–1320.
LIN et al.
Conclusion: This study demonstrates the validity and responsiveness of the
CRS-PRO in subjects receiving ESS.
KEYWORDS
chronic rhinosinusitis, endoscopic sinus surgery, patient-reported outcome measure, SNOT-22,
CRS-PRO
INTRODUCTION
Chronic rhinosinusitis (CRS) is a common chronic
inflammatory condition of the nose and paranasal sinuses.
In addition to economic burden, CRS patients also suffer
from impairment in several domains of health status
including fatigue, pain, and sleep.1,2 CRS comprises two
major phenotypes: CRS with nasal polyps (CRSwNP) and
CRS without nasal polyps (CRSsNP) based on intranasal
evaluation, although it is unclear if they are pathogenically
distinct.3 However, most clinical trials of CRS have been
isolated to patients of a single phenotype.4,5
Patient-reported outcome measurements (PROMs)
measure various aspects of the patient’s experience with a
disease and are increasingly used as endpoints in clinical
trials.6 CRS-specific PROMs like the 22-item Sino-Nasal
Outcome Test (SNOT-22) and chronic rhinosinusitis
patient-reported outcome (CRS-PRO) measure comprise
symptom items common among CRS patients but may
include salient general health items. General health
PROMs measure physical, mental, and social health con-
cepts seen more broadly in chronic disease, including CRS.
For instance, the National Institutes of Health (NIH)’s
Patient Reported Outcome Measurement Information Sys-
tem (PROMIS) provides item banks for the measurement
of generic symptoms and functional reports,6,7 which have
been shown to reflect increased impairment due to fatigue,
sleep disturbance, and pain in the CRS population.8 The
SNOT-22 is the most widely used CRS-specific PROM
within the specialty and has well-recognized psychometric
properties among CRS patients. However, examining the
development of the SNOT-22 reveals multiple issues. The
Rhinosinusitis Outcome Measure 31 (RSOM-31), from
which items in the SNOT-22 were selected, was developed
with input from patients who did not necessarily meet
current definitions of CRS; for instance only three patients
had CRSwNP, more patients in the sample had rhinitis
(82%) than chronic sinusitis (41%), and the 55 patients
with CRS were diagnosed with symptoms inconsistent
with current criteria.9 Subsequent modifications that
resulted in the 20-item SNOT (SNOT-20) and SNOT-22
were physician-driven, with an overall lack of input from
CRS patients.10,11
1309
Although some very recent clinical trials in CRSwNP
have reported the SNOT-22 as a secondary endpoint,
no CRS-specific PROM is registered in the U.S. Food
and Drug Administration (FDA)’s most recent PROM
compendium.12 In guidance from the FDA on acceptable
PROMs,13 an acceptable PROM for labeling claims requires
clearly documented input from the target populations—
in this case, CRSwNP and CRSsNP patients. Furthermore,
when considering the acceptability of an existing PROM,
a “key consideration” is whether the population in the
clinical study is similar to or different from that used
to develop the PRO instrument. Modifications of existing
PRO instruments also require patient involvement with
cognitive interviews and item development.14 To overcome
the developmental limitations of the SNOT-22 and its pre-
decessors, Ghadersohi et al.15,16 described the develop-
ment of the CRS-PRO instrument, which extensively uti-
lized patient input from patients with well-characterized
CRSsNP and CRSwNP and included both patients who did
or did not have prior sinus surgery to capture the full spec-
trum of disease severity. The CRS-PRO was subsequently
shown to be valid and responsive in assessing symptom
changes in medically treated CRS patients. In these prior
studies, the validated version of the CRS-PRO was a 12-item
instrument scored as a single total score. However, disease-
specific instruments frequently are composed of multi-
ple subdomains reflecting the major domains of health
affected by a condition. Therefore, this study examines the
factor structure, as well as validity and responsiveness of
the CRS-PRO in subjects receiving ESS.
PATIENTS AND METHODS
Patients
All data were obtained from the Northwestern CRS subject
registry; patients were recruited from 2017 to 2019, prior to
undergoing ESS, at Northwestern Medical Group Depart-
ment of Otolaryngology–Head and Neck Surgery, a tertiary
care referral center. Extent of ESS was not standardized
and was based on what was clinically indicated for each
individual patient. All patients were age 18 to 89 years at
time of enrollment and had a diagnosis of CRS. Exclusion
1310
criteria included limited English comprehension, diagno-
sis of recurrent acute sinusitis, and pregnancy. The proto-
col for the CRS subject registry was reviewed and approved
by the Northwestern Institutional Review Board.
We performed analyses on the 111 CRS patients who
prospectively completed questionnaires prior to ESS and
at 3-month follow-up, as well as 86 patients who had 6-
month follow-up after ESS. The following information was
extracted from the baseline visit: age, gender, race, CRS
subtype (with/without polyps), asthma, allergies, aspirin
sensitivity, and smoking status. Patients had also com-
pleted the 12-item CRS-PRO, SNOT-22, and four PROMIS
short forms: Fatigue (version 6a), Sleep Disturbance
(version 6a), Pain Interference (version 6a), and Satisfac-
tion with Participation in Discretionary Social Activities
(version 7a) that had previously been identified to rep-
resent important general health concepts mentioned by
patients with CRS.8
For the CRS-PRO, each of the 12 items is scored 0 to 4,
with 4 representing the most severe degree of symptoms
(maximum score 48). The SNOT-22 is composed of 22
items, each scored from 0 to 5, with 5 representing the most
severe degree of symptoms (maximum score 110). Prior
studies of the SNOT-22 have generally described four sub-
domains with the general categorization of rhinological
symptoms, otologic and/or facial symptoms, sleep-related
symptoms, and psychological symptoms.17–19 There has
been some variation with categorization of the SNOT-22
items into the Sleep versus Psychological subdomains;
we chose the SNOT-22 subdomain structure used in the
majority of prior studies.17–19 The PROMIS short forms
are six to seven items long and are scaled to t scores; a
score of 50 represents the mean of the general population
and a change of 10 represents a change of 1 standard
deviation. Study data were collected and managed using
REDCap electronic data capture tools hosted at North-
western University Feinberg School of Medicine.20,21 Any
missing questionnaire items (0.32%) of item answers, were
imputed based on the mean of all available responses for
the specific questionnaire item.22 Missing objective data
(sinus computed tomography [CT] scan score, endoscopy
score) were obtained by chart review (1.8% of participants).
At the last pre-ESS clinic assessment, patients had a CT
scan and nasal endoscopy. CT scans were scored based on
the Lund-Mackay score, in which a higher score indicates
greater extent of inflammation.23 Endoscopy was quanti-
fied by the Modified Lund-Kennedy score, which grades
disease severity in terms of edema, discharge, and polyps in
the nasal cavity (each from zero to two points for each side;
total possible score of 12 points).24 A higher total score indi-
cates greater extent of disease seen on endoscopic exam.
CT scan and endoscopy were not completed at the 3-month
and 6-month assessments.
CRS-PRO responsiveness and validity in ESS
Factor analysis
We conducted an exploratory principal axis factor analysis
(EFA) on the 12 items of the CRS-PRO. A Kaiser-Meyer-
Olkin (KMO) test was performed to determine dataset suit-
ability for EFA; higher KMO values indicate sampling ade-
quacy for each variable in the model, with values >0.50
considered acceptable.25 Bartlett’s test of sphericity was
also performed to assess for adequate collinearity (p < 0.05)
between variables. Two methods were used to determine
the number of factors with significant contributions: (1)
scree plot and (2) Eigenvalue cutoff of 1.0 for factors. Vari-
able loading onto each factor was determined with a vari-
max rotation (an orthogonal method, valid for use with
uncorrelated factors) on the original EFA. Cutoff for signif-
icant variable loading onto a given factor was set at 0.40.26
Convergent validity
We calculated the correlations between the changes in
the CRS-PRO score and changes in the (1) SNOT-22 and
(2) PROMIS short forms. No objective measures (CT,
endoscopy scores) were available at any of the follow-up
time points for the CRS Registry surgical cohort. There-
fore, the convergent validity between the CRS-PRO and
objective measures could only be examined in terms of
correlation at baseline. Spearman’s correlation was used
for pairwise correlations as data were nonparametric. For
correlation coefficients, we considered ≥0.1 as poor, ≥0.3
as fair, ≥0.6 as moderate, and ≥0.8 as very strong.27
Responsiveness
The statistical significance of the changes between base-
line and follow-up of the CRS-PRO, SNOT-22, and PROMIS
short forms was evaluated using a Wilcoxon rank-sum test
(for nonparametric data). Effect sizes were estimated using
the Cohen’s d (magnitude of change between baseline and
follow-up for a given instrument or subdomain, divided by
its baseline standard deviation). For effect size interpreta-
tion, we considered ≥0.2 as small, ≥0.5 as moderate, ≥0.8
as large, and ≥1.20 as very large.28
Clinically important differences
The clinically important difference (CID) represents
a meaningful change in health status as perceived by
patients.29 We computed multiple CID estimates for
the CRS-PRO by distribution-based and anchor-based
approaches, using data from patients with 3-month
LIN et al.
follow-up. For the distribution-based approaches, we
used a standard deviation (SD)-based approach, which
defines the CID as 0.5*SD,30 where the SD is that of the
baseline CRS-PRO in the sample of n = 111 patients.
For the anchor-based methods, we used the previously
determined minimal CIDs (MCIDs) of the (1) SNOT-22
(reported to be 8.9)11 and (2) PROMIS Fatigue (estimated
at 4.0),31 and calculated the corresponding CID of the
CRS-PRO using a linear regression.
Statistical software
All statistical analyses were performed using JMP (SAS
Institute, Marlow, Buckinghamshire, UK), R (R Founda-
tion for Statistical Computing, Auckland, New Zealand),
or SPSS (IBM Corporation, Armonk, NY). Demographic
and clinical characteristics were reported as frequencies
and percentages for categorical variables; means, SDs, and
range for continuous variables. Significance was deter-
mined at p < 0.05.
RESULTS
Demographics
In total, 111 patients completed follow-up at 3 months and
86 patients at 6 months. Table 1 summarizes the baseline
demographics. There were some significant though antic-
ipated differences in the comorbidities and demographics
for the CRSsNP and CRSwNP phenotypes, consistent with
prior studies.16,32–34 Baseline mean Lund-Mackay CT score
for the group of patients with 3-month follow-up was 12.8
± 4.8. Baseline mean modified Lund-Kennedy endoscopic
score for this group was 3.9 ± 3.1. CRSwNP patients had a
greater CT score (15.4 ± 4.7) than those with CRSsNP (10.6
± 3.7) and greater endoscopy score (5.1 ± 3.0 in CRSwNP,
2.9 ± 2.8 in CRSsNP), which was consistent with prior
studies.16
Factor analysis
Dot-plots of loadings for each CRS-PRO item and scree
plot are shown in Figure 1. EFA suggested that three fac-
tors (explaining 64.8% of the CRS-PRO score variance)
described the most prominent symptom clustering pat-
terns. The KMO value for the EFA was 0.781 and Bartlett’s
test value was p < 0.0001, which are considered adequate.
The relative loadings of CRS-PRO symptoms on each
factor revealed patterns of symptoms. We termed these
1311
factors the Rhino-Psychologic, Facial Discomfort, and
Cough symptom clusters, according to the magnitude of
loadings on each factor and the concepts raised in items
within each cluster. Each CRS-PRO item loaded most
strongly on one factor alone, with loadings ranging from
0.449 to 0.787 for Rhino-Psychologic, 0.790 to 0.966 for
Facial Discomfort, and 0.598 to 0.751 for Cough. Although
the scree plot shows four factors above a threshold of eigen-
value > 1, factor 4 had a borderline eigenvalue of 1.03, and
the first break in the scree plot (not counting the break after
factor 1, which is almost always present)35 occurred after
factor 3. As the eigenvalue > 1 criteria often results in too
many factors being retained,35 and accounting for the bor-
derline factor 4 eigenvalue, we used the breakpoint method
instead to choose three factors for extraction. Of the 12
items in the CRS-PRO, 67% (eight items) were in the Rhino-
Psychologic factor whereas the Facial Discomfort and
Cough factors each accounted for 17% (two items each).
Convergent validity
We assessed the convergent validity of the CRS-PRO
instrument by correlating the baseline to 3-month change
in CRS-PRO mean score after ESS to the change in (1)
SNOT-22 and (2) PROMIS short forms (Table 2). There
was a moderate, highly significant correlation between the
SNOT-22 and CRS-PRO (0.792, p < 0.0001). The CRS-PRO
also had significant fair correlations with the changes in
PROMIS Fatigue (0.508, p < 0.0001) and Sleep Disturbance
(0.518, p < 0.0001) and a modest but significant correla-
tion with PROMIS Pain (0.236, p < 0.05). Neither the CRS-
PRO nor the SNOT-22 had significant correlation with the
PROMIS Satisfaction with Participation in Discretionary
Social Activities (SPDSA).
All three CRS-PRO factors correlated significantly with
the SNOT-22 total score, but most strongly the CRS-
PRO Rhino-Psychologic factor. The Rhino-Psychologic fac-
tor also had fair correlations with PROMIS Fatigue and
PROMIS Sleep Disturbance compared to the other two fac-
tors. Out of the three CRS-PRO factors, only Facial Discom-
fort had significant correlation with PROMIS Pain. Com-
paring the correlations for the CRS-PRO factors and SNOT-
22 subdomains, no SNOT-22 subdomains had more than
a poor to fair correlation with any of the PROMIS mea-
sures. SNOT-22 Ear/Face and SNOT-22 Psychological had
significant correlations with PROMIS Pain only. Neither
the CRS-PRO factors nor SNOT-22 subdomains correlated
with PROMIS SPDSA.
We also assessed the convergent validity of the CRS-
PRO by correlating the baseline CRS-PRO with the
presurgical endoscopic and CT objective scores (Table 3).
 1312

F I G U R E 1 Dot-plot of loadings for each CRS-PRO survey item for: (A) factor 1, representing the “rhino-psychologic” symptoms subset; (B) factor 2, representing the “facial discomfort”
symptoms subset; and (C) factor 3, representing the “cough” symptoms subset, as well as the scree plot of eigenvalues for these factors. Abbreviation: CRS-PRO, chronic rhinosinusitis
patient-reported outcome measure.
CRS-PRO responsiveness and validity in ESS
LIN et al. 1313

TA B L E 1 Baseline demographic information and objective scores, for subjects with 3-month follow-up†
Demographics All patients CRSsNP CRSwNP
N (%) 111 (100) 60 (54.1) 51 (45.9)
Age (years) 46.2 ± 14.9 (19–76) 46.5 ± 16.2 (19–74) 45.8 ± 13.3 (26–76)
Race, n (%)
White 77 (69.4) 46 (76.7) 31 (60.8)
African American 2 (1.8) 0 (0.0) 2 (3.9)
Asian 8 (7.2) 4 (6.7) 4 (7.8)
Unknown 24 (21.6) 10 (16.7) 14 (27.5)
Gender, n (%)
Female 55 (49.6) 37 (61.7) 18 (35.3)**
Male 56 (50.4) 23 (38.3) 33 (64.7)
Previous ESS, n (%) 46 (41.4) 21 (35.0) 25 (49.0)
Asthma, n (%) 44 (39.6) 17 (28.3) 27 (52.9)**
AERD, n (%) 8 (7.2) 1 (1.7) 7 (13.7)*
Allergies, n (%) 50 (45.0) 27 (45.0) 23 (45.1)
Smoking status, n (%)
Current 2 (1.8) 1 (1.7) 1 (2.0)
Former 28 (25.2) 15 (25.0) 13 (25.5)
Never 81 (73.0) 44 (73.3) 37 (72.5)
Lund-Mackay score 12.8 ± 4.8 (CI, 11.9–13.7) 10.6 ± 3.7 (CI, 9.6–11.5) 15.4 ± 4.7*** (CI, 14.1–16.7)
Endoscopy scoring
MLK total 3.9 ± 3.1 (CI, 3.3–4.5) 2.9 ± 2.8 (CI, 2.2–3.6) 5.1 ± 3.0** (CI, 4.2–5.9)
Edema 1.3 ± 1.5 (CI, 1.0–1.6) 1.2 ± 1.4 (CI, 0.8–1.6) 1.3 ± 1.6 (CI, 0.8–1.8)
Discharge 1.5 ± 1.7 (CI, 1.2–1.8) 1.7 ± 1.8 (CI, 1.2–2.1) 1.3 ± 1.6 (CI, 0.9–1.8)
Polyps (N = 51) 2.4 ± 1.0 (CI, 2.1–2.7) N/A 2.4 ± 1.0 (CI, 2.1–2.7)
Baseline CRS-PRO 23.8 ± 9.9 (CI, 21.9–25.7) 24.5 ± 9.4 (CI, 22.0–26.9) 23.0 ± 10.6 (CI, 20.0–26.0)
Baseline SNOT-22 42.3 ± 19.9 (CI, 38.5–46.0) 45.0 ± 20.8 (CI, 39.6–50.4) 39.1 ± 18.4 (CI, 33.9–44.2)
Baseline PROMIS
Fatigue 50.6 ± 10.6 (CI, 48.6–52.6) 52.8 ± 10.8 (CI, 50.0–55.6) 48.1 ± 10.0 (CI, 45.2–50.9)
Sleep disturbance 52.2 ± 9.5 (CI, 50.4–54.0) 53.9 ± 9.4 (CI, 51.4–56.3) 50.2 ± 9.2 (CI, 47.6–52.8)
SPDSA 52.2 ± 8.9 (CI, 50.5–53.8) 51.2 ± 8.6 (CI, 48.9–53.4) 53.3 ± 9.1 (CI, 50.8–55.9)
Pain interference 49.3 ± 8.3 (CI, 47.7–50.8) 51.1 ± 8.5 (CI, 48.9–53.3) 47.2 ± 7.7 (CI, 45.0–49.3)
†
Values are mean ± SD (range) or n (%). PROMIS instruments scored based on standardized t scores. Lund-Mackay CT score, maximum 24 points. MLK total,
maximum 12 points. CRS-PRO, maximum 12 points. SNOT-22, maximum 110 points.
Abbreviations: AERD, aspirin-exacerbated respiratory disease; CI, confidence interval; CRS, chronic rhinosinusitis; CRSsNP, CRS without nasal polyps; CRSwNP,
CRS with nasal polyps; CRS-PRO, CRS patient-reported outcome; CT, computed tomography; ESS, endoscopic sinus surgery; MLK, modified Lund-Kennedy; N/A,
not applicable; PROMIS, Patient-Reported Outcomes Measurement Information System; SNOT-22, 22-item Sino-Nasal Outcome Test; SPDSA, satisfaction with
participation in discretionary social activities.
*p < 0.05.
**p < 0.01.
***p < 0.001.

Interestingly, the CRS-PRO cough factor had a poor but
significant correlation with endoscopic discharge sever-
ity (0.260, p < 0.01). The CRS-PRO Facial Discomfort fac-
tor and endoscopic edema severity had a poor but sig-
nificant negative correlation (−0.193, p < 0.05). How-
ever, the CRS-PRO total did not correlate well with any
objective measures when considered across all patients,
though had a fairly significant correlation (0.319, p < 0.05)
with polyp score among patients with CRSwNP. The
CRS-PRO Rhino-Psychologic factor and Lund-Kennedy
score also had a significant correlation among CRSwNP
patients. There was a similar overall lack of correlation
between the SNOT-22 total and subdomains with objec-
tive scores, and no significant correlations between base-
line objective measures and any of the general health
domains.
1314 CRS-PRO responsiveness and validity in ESS

Responsiveness

Abbreviations: CRS-PRO, chronic rhinosinusitis patient-reported outcome; PROMIS, Patient-Reported Outcomes Measurement Information System; SNOT-22, 22-item Sino-Nasal Outcome Test; SPDSA, satisfaction
T A B L E 2 Correlation of the change in instrument and subdomain score to change in general patient reported outcome measure scores (subjective scores), all patients with 3-month

Responsiveness data is presented in Table 4. The CRS-PRO

PROMIS SPDSA total had a statistically significant decrease from 23.8 ± 9.9
at baseline to 9.5 ± 8.0 at 3-month follow-up (p < 0.0001)
with a very large effect size of 1.44 (CI, 1.14 to 1.73), which
−0.046
−0.042

−0.037
−0.019

−0.164
−0.181

−0.118
was similar to the SNOT-22 total score effect size of 1.41

0.006
0.077 (CI, 1.11 to 1.70). For the group with 3-month data, the
effect sizes for all the CRS-PRO factors were moderate to
very large, and all changes were highly statistically sig-
nificant. The CRS-PRO total and Rhino-Psychologic fac-
tor also maintained very large effect sizes at 6-month
PROMIS pain

follow-up. Comparing the performance of the CRS-PRO to

interference

the SNOT-22, both measures demonstrated similarly large

0.436***

0.323**

0.258**

effect sizes at 3 and 6 months after ESS. Interestingly, the

0.236*

0.214*

0.193*
0.087
0.149
0.147

CRS-PRO was more responsive than its individual factors

whereas the SNOT-22 demonstrated the largest effect size
for its Rhinological subdomain.
All of the PROMIS measures had moderate effect sizes,
smaller than the CRS-PRO. The SPDSA effect size was neg-
ative because higher scores are indicative of greater social
PROMIS sleep
disturbance

satisfaction; hence an increased post-ESS score is expected.

At baseline, the CRSsNP patients had higher CRS-PRO
0.458***

0.376***
0.514***
0.518***

0.260**

0.298**
0.265**

0.103

total scores and Facial Discomfort, and Cough factor scores

0.172

than CRSwNP patients (Table 5). Changes between pre-

ESS and 3-month data and associated effect sizes were also
calculated separately for CRSsNP and CRSwNP patients.
Over a follow-up period of 3 months, the CRS-PRO total
PROMIS fatigue

was more responsive in the CRSwNP group (ES 1.49; CI,

1.05 to 1.94) than the CRSsNP group (ES 1.38; CI, 0.98 to
1.78), which was also observed using the SNOT-22. In both
0.364***
0.508***

0.508***

0.458***
0.551***
0.303**

0.218*

CRSwNP and CRSsNP patients, very large effect sizes were

0.178
0.121

observed in the CRS-PRO Rhino-Psychologic domain and

moderate to large effect sizes were observed in the Facial
Discomfort and Cough factors. The responsiveness of the
CRS-PRO at 6 months was largely similar to the patterns
observed with the 3-month data.
SNOT-22

0.450***
0.634***
0.459***

0.645***
0.792***

0.827***
0.519***
0.761***

CID
1

with participation in discretionary social activities.

We estimated the value of the CRS-PRO CID using the

distribution-based approach of 0.5*SD, and found a CID
of 5.0 based on this method (Table 6). Using anchor-based
approaches, we found a difference of 5.0 using the SNOT-
22 MCID as an anchor, and 7.5 using PROMIS Fatigue as
an anchor. Therefore, estimates of the CID of the CRS-PRO
Rhino-psychologic
Facial discomfort

established using multiple methods is between 5.0 and 7.5,

follow-up (N = 111)

Psychological
Rhinological
CRS-PRO total

and we thus propose 6.0 as the instrument’s CID.

SNOT-22 total
Parameter

Ear/face

***p < 0.0001.

Cough

Sleep

**p < 0.01.

*p < 0.05.
 LIN et al.

TA B L E 3 Correlation coefficients of the baseline instrument and subdomain score to presurgical objective measure scores for patients with 3-month follow-up
Endoscopic scores
Total LK CRSwNP
Parameter CT LM Total LK (N = 111) (N = 51) Polyp (N = 51) Discharge Edema
CRS-PRO total 0.094 0.117 0.203 0.319* 0.077 0.074
Rhino-psychologic 0.138 0.174 0.293* 0.230 0.052 0.157
Facial discomfort −0.165 −0.147 0.136 −0.082 0.014 −0.193*
Cough −0.025 0.082 0.012 0.0774 0.260** 0.063
SNOT-22 total −0.005 0.071 0.185 0.347* 0.101 0.077
Rhinological 0.196* 0.235 0.268 0.466** 0.184 0.075
Ear/face −0.154 −0.133 −0.063 0.201 −0.019 −0.152
Sleep −0.084 0.002 0.097 −0.027 0.055 0.127
Psychological 0.155 0.229* 0.301* 0.220 0.186 0.185
PROMIS short forms
Fatigue −0.072 −0.105 0.060 −0.002 −0.015 0.039
Sleep disturbance −0.049 −0.074 0.068 −0.019 −0.095 0.163
SPDSA −0.017 −0.0033 −0.035 0.184 −0.075 −0.083
Pain interference −0.081 −0.064 0.030 0.137 0.060 −0.020
Abbreviations: CRS, chronic rhinosinusitis; CRS-PRO, CRS patient-reported outcome; CRSwNP, CRS with nasal polyps; CT LM, Lund-Mackay computed tomography score; LK, Lund-Kennedy score; PROMIS, Patient-
Reported Outcomes Measurement Information System; SNOT-22, 22-item Sino-Nasal Outcome Test; SPDSA, satisfaction with participation in discretionary social activities.
*p < 0.05.
**p < 0.01.
***p < 0.0001.
1315
1316 CRS-PRO responsiveness and validity in ESS

TA B L E 4 Responsiveness at 3 and 6 months of CRS-PRO, SNOT-22, and PROMIS with subdomains for overall cohort†
Baseline Mean ± SD 3 Months Mean ± SD 3-Month ES Cohen’s d 6-Month ESa Cohen’s
Parameter (CI) (CI) (CI) d (CI)
CRS-PRO total 23.8 ± 9.9 (CI, 21.9–25.7) 9.5 ± 8.0 (CI, 8.0–11.0) 1.44*** (CI, 1.14–1.73) 1.28*** (CI, 0.95–1.61)
Rhino-psychologic 16.7 ± 7.5 (CI, 15.3–18.2) 6.7 ± 5.7 (CI, 5.6–7.8) 1.33*** (CI, 1.04–1.63) 1.21** (CI, 0.88–1.54)
Facial discomfort 3.5 ± 2.6 (CI, 3.0–4.0) 0.9 ± 1.4 (CI, 0.6–1.1) 1.01*** (CI, 0.73–1.29) 0.74*** (CI, 0.43–1.05)
Cough 3.5 ± 2.4 (CI, 3.1–4.0) 1.9 ± 2.1 (CI, 1.6–2.3) 0.67*** (CI, 0.40–0.94) 0.47** (CI, 0.16–0.77)
SNOT-22 total 42.3 ± 19.9 (CI, 14.3 ± 13.1 (CI, 11.9–16.8) 1.41*** (CI, 1.11–1.70) 1.26*** (CI, 0.93–1.58)
38.5–46.0)
Rhinological 19.8 ± 7.6 (CI, 18.4–21.2) 7.9 ± 6.9 (CI, 6.6–9.2) 1.58*** (CI, 1.27–1.88) 1.59*** (CI, 1.24–1.93)
Ear/pain 6.1 ± 4.5 (CI, 5.2–6.9) 1.9 ± 2.7 (CI, 1.4–2.4) 0.93*** (CI, 0.65–1.21) 0.65*** (CI, 0.34–0.96)
Sleep 14.8 ± 10.6 (CI, 12.9–16.8) 4.2 ± 5.5 (CI, 3.2–5.3) 1.00*** (CI, 0.73–1.29) 0.84*** (CI, 0.52–1.15)
Psychological 1.6 ± 1.9 (CI, 1.2–1.9) 0.33 ± 0.83 (CI, 0.2–0.5) 0.67*** (CI, 0.39–0.94) 0.54*** (CI, 0.23–0.84)
PROMIS short forms
Fatigue 50.6 ± 10.6 (CI, 42.6 ± 8.6 (CI, 41.0–44.2) 0.76*** (CI, 0.48–1.03) 0.62*** (CI, 0.31–0.92)
48.6–52.6)
Sleep disturbance 52.2 ± 9.5 (CI, 50.4–54.0) 45.8 ± 7.7 (CI, 44.4–47.3) 0.67*** (CI, 0.40–0.94) 0.51** (CI, 0.21–0.82)
SPDSA 52.2 ± 8.9 (CI, 50.5–53.8) 57. 3 ± 9.9 (CI, 55.4–59.2) −0.58*** (CI, −0.85 to −0.57** (CI, −0.88 to
−0.31) −0.26
Pain interference 49.3 ± 8.3 (CI, 47.7–50.8) 43.3 ± 4.9 (CI, 42.4–44.2) 0.72*** (CI, 0.45–0.99) 0.56*** (CI, 0.26–0.87)
†
12-Item CRS-PRO score range 0–48, total SNOT-22 score range 0–110, SNOT rhinological score range 0–40, SNOT ear/face score range 0–20, SNOT sleep score
range 0–40, SNOT psychological score range 0–10. PROMIS instruments are scored based on standardized t scores.
Abbreviations: CI, confidence interval; CRS-PRO, chronic rhinosinusitis patient-reported outcome; ES, effect size; PROMIS, Patient-Reported Outcomes Measure-
ment Information System; SNOT-22, 22-item Sino-Nasal Outcome Test; SPDSA, satisfaction with participation in discretionary social activities.
a
6-Month ES was calculated using corresponding patient baseline and 6-month data.
*p < 0.05.
**p < 0.01.
***p < 0.0001.

DISCUSSION The factor analysis finds the cardinal symptoms of CRS

In this study, we evaluated and found three robust fac-
tors within the CRS-PRO, which included the Rhino-
Psychologic, Facial Discomfort, and Cough factors, and
propose these as subdomains for the instrument. Most
items loaded strongly on one factor alone. The over-
all CRS-PRO score and Rhino-Psychologic component
demonstrated moderate to strong convergent validity with
PROMs such as the SNOT-22, and validated measures of
general health, especially the PROMIS Fatigue and Sleep
disturbance scales. Though the correlations between the
CRS-PRO and presurgical objective measures were not
strong, some associations were noted between each of
the three factors and components of endoscopic sever-
ity measures. The CRS-PRO and all the extracted factors
were responsive to ESS, as shown by medium-to-large
effect sizes for all the factors. In particular, the Rhino-
Psychologic factor was associated with the largest effect
sizes, that were further magnified in the CRSwNP group.
These findings demonstrate the validity of the CRS-PRO
for measuring treatment-related changes following ESS,
with performance characteristics as favorable as the longer
SNOT-22 instrument.
distributed into all the three factors and reflects some
departures from the SNOT-22 factor structure where the
core symptoms load into two of the four subdomains.
Our first factor, the Rhino-Psychologic factor, interestingly
loaded eight items from distinct rhinologic, psychologic,
and sleep-related concepts. Their assembly into the most
heavily weighted factor suggests these items may reflect
a latent aggregation of these seemingly disparate patient
impairments. This differs from the SNOT-22 subdomains,
which usually find the sleep subdomain as its first factor
and the rhinological and psychological items forming sep-
arate subdomains.17–19 The cardinal symptoms of CRS typ-
ically load only in the Rhinological and Ear Pain subdo-
mains of the SNOT-22. In contrast, the CRS-PRO has the
cardinal symptoms distributed into all the factors with fac-
tor 1 (“Rhino-psychologic”) capturing the nasal obstruc-
tive, discharge, and smell symptoms; factor 2 (“Facial Dis-
comfort”) is composed, intuitively, of “pressure in the face”
and “face hurts” items and factor 3 (“Cough”) is composed
of “cough” and “trouble clearing mucus from the throat”
items, which relate to postnasal drip and associated cough.
We continue to find strong convergent validity of the
SNOT-22 and the CRS-PRO and its factors. At 3 months,
LIN et al. 1317

T A B L E 5 Responsiveness assessment: comparison of baseline and follow-up scores and effect sizes of total instrument and subdomain
scores of the CRS-PRO measures for CRSwNP patients at 3 months (N = 51) and 6 months (N = 39), and for CRSsNP patients at 3 months (N
= 60) and 6 months (N = 47)
Baseline Mean ± SD 3 Months Mean ± SD 3-Month ES Cohen’s d 6-Month ESa Cohen’s
Parameter (CI) (CI) (CI) d (CI)
CRSwNP patients
CRS-PRO total 23.0 ± 10.6 (CI, 7.2 ± 6.8 (CI, 5.2–9.1) 1.49*** (CI, 1.05–1.94) 1.26*** (CI, 0.77–1.76)
20.0–26.0)
Rhino-psychologic 17.1 ± 8.0 (CI, 14.9–19.4) 5.4 ± 2.3 (CI, 3.9–6.9) 1.46*** (CI, 1.02–1.91 ) 1.28*** (CI, 0.79–1.78)
Facial discomfort 2.9 ± 2.4 (CI, 2.3–3.6) 0.5 ± 1.2 (CI, 0.1–0.8) 1.01*** (CI, 0.60–1.43) 0.64** (CI, 0.18–1.10)
Cough 2.9 ± 2.5 (CI, 2.2–3.6) 1.3 ± 1.7 (CI, 0.8–1.8) 0.65** (CI, 0.25–1.06) 0.52* (CI, 0.06–0.97)
SNOT-22 total 39.1 ± 18.4 (CI, 33.9–44.2) 8.8 ± 9.6 (CI, 6.1–11.5) 1.65*** (CI, 1.19–2.10) 1.24*** (CI, 0.75–1.74)
Rhinological 20.0 ± 8.2 (CI, 17.7–22.3) 5.1 ± 5.4 (CI, 3.5–6.6) 1.83*** (CI, 1.36–2.30) 1.53*** (CI, 1.02–2.04)
Ear/pain 5.1 ± 4.4 (CI, 3.9−6.3) 1.2 ± 2.3 (CI, 0.5–1.9) 0.88*** (CI, 0.47–1.29) 0.54** (CI, 0.08–1.00)
Sleep 12.5 ± 9.5 (CI, 9.8–15.1) 2.4 ± 4.3 (CI, 1.2–3.6) 1.06*** (CI, 0.64–1.48) 0.82** (CI, 0.35–1.29)
Psychological 1.5 ± 1.8 (CI, 1.0–2.0) 0.2 ± 0.5 (CI, 0.0–0.3) 0.76*** (CI, 0.35–1.16) 0.56** (CI, 0.10–1.02)
PROMIS forms
Fatigue 48.1 ± 10.0 (CI, 39.4 ± 8.0 (CI, 37.2–41.7) 0.86*** (CI, 0.45–1.27) 0.70** (CI, 0.23–1.16)
45.2–50.9)
Sleep disturbance 50.2 ± 9.2 (CI, 47.6–52.8) 43.2 ± 8.1 (CI, 40.9−45.5) 0.76*** (CI, 0.35–1.17) 0.50* (CI, 0.05–0.96)
SPDSA 53.3 ± 9.1 (CI, 50.8−55.9) 58.8 ± 10.6 (CI, −0.60** (CI, −1.0 to −0.60** (CI, −1.06 to
55.8−61.8) −0.2) −0.14)
Pain interference 47.2 ± 7.7 (CI, 45.0–49.3) 42.1 ± 3.4 (CI, 41.1–43.0) 0.67*** (CI, 0.26–1.07) 0.50** (CI, 0.04–0.96)
CRSsNP patients
CRS-PRO total 24.5 ± 9.4 (CI, 22.0–26.9) 11.5 ± 8.4 (CI, 9.4–13.7) 1.38*** (CI, 0.98–1.78) 1.30*** (CI, 0.85–1.75)
Rhino-psychologic 16.4 ± 7.1 (CI, 14.6–18.3) 7.8 ± 5.9 (CI, 6.3−9.3) 1.21*** (CI, 0.81–1.60) 1.03*** (CI, 0.60–1.47)
Facial discomfort 4.0 ± 2.7 (CI, 3.3–4.7) 1.2 ± 1.6 (CI, 0.8–1.6) 1.04*** (CI, 0.65–1.42) 0.82*** (CI, 0.39–1.24)
Cough 4.1 ± 2.1 (CI, 3.5–4.6) 2.5 ± 2.2 (CI, 2.0–3.1) 0.73** (CI, 0.36–1.10) 0.43* (CI, 0.02−0.84)
SNOT-22 total 45.0 ± 20.8 (CI, 19.0 ± 13.8 (CI, 15.4–22.6) 1.25*** (CI, 0.85–1.64) 1.27*** (CI, 0.82–1.72)
39.6–50.4)
Rhinological 19.6 ± 7.1 (CI, 17.8–21.4) 10.3 ± 7.1 (CI, 8.5–12.1) 1.32*** (CI, 0.92–1.72) 1.68*** (CI, 1.20–2.16)
Ear/pain 6.9 ± 4.5 (CI, 5.7–8.0) 2.4 ± 2.8 (CI, 1.7–3.2) 0.99*** (CI, 0.61–1.37) 0.74*** (CI, 0.32–1.17)
Sleep 16.9 ± 11.1 (CI, 14.0–19.7) 5.8 ± 6.0 (CI, 4.2–7.3) 1.00*** (CI, 0.62–1.39) 0.85*** (CI, 0.43–1.28)
Psychological 1.6 ± 1.9 (CI, 1.1–2.1) 0.5 ± 1.0 (CI, 0.2–0.7) 0.59** (CI, 0.22−0.96) 0.52* (CI, 0.10−0.93)
PROMIS forms
Fatigue 52.8 ± 10.8 (CI, 50.0–55.6) 45.2 ± 8.2 (CI, 43.1–47.4) 0.70*** (CI, 0.33–1.08) 0.54** (CI, 0.13–0.96)
Sleep disturbance 53.9 ± 9.4 (CI, 51.4–56.3) 48.1 ± 6.5 (CI, 46.4–49.8) 0.61** (CI, 0.24–0.98) 0.53* (CI, 0.12–0.95)
SPDSA 51.2 ± 8.6 (CI, 48.9–53.4) 56.0 ± 9.2 (CI, 53.6–58.4) −0.56*** (CI, −0.93 to −0.54* (CI, −0.95 to
−0.19) −0.12)
Pain interference 51.1 ± 8.5 (CI, 48.9–53.3) 44.3 ± 5.8 (CI, 42.8–45.8) 0.79** (CI, 0.42–1.17) 0.62** (CI, 0.20–1.04)
12-item CRS-PRO score range 0–48, total SNOT-22 score range 0−110, SNOT Rhinological score range 0−40, SNOT ear/face score range 0–20, SNOT sleep score
range 0–40, SNOT psychological score range 0–10. PROMIS instruments are scored based on standardized t scores.
Abbreviations: CI, confidence interval; CRS-PRO, chronic rhinosinusitis patient-reported outcome; ES, effect size; PROMIS, Patient-Reported Outcomes Measure-
ment Information System; SNOT−22 = 22-item Sino-Nasal Outcome Test; SPDSA, satisfaction with participation in discretionary social activities.
a
6-month ES was calculated using corresponding patient baseline and 6−month data.
*p < 0.05.
**p < 0.01.
***p < 0.0001.
1318
TA B L E 6 Estimates of the CID for the CRS-PRO
Method Value
Distribution-based methods
0.5*SD 5.0
Anchor-based methods
SNOT-22 (MCID) 5.0
PROMIS fatigue (MCID) 7.5
Abbreviations: CID, clinically important difference; CRS-PRO, chronic rhinos-
inusitis patient-reported outcome; MCID, minimal clinically important differ-
ence; PROMIS, Patient-Reported Outcomes Measurement Information Sys-
tem; SNOT−22 = 22-item Sino-Nasal Outcome Test.
*p < 0.05.
**p < 0.01.
***p < 0.0001.
the change in the CRS-PRO and SNOT-22 had a strong
correlation of 0.792. The SNOT-22 has been validated as
a measure of symptom change following ESS,2 and most
recently in clinical trials of biologics,4,5 therefore demon-
strating the validity of the CRS-PRO for measuring these
changes. The CRS-PRO also correlated moderately well
with changes in three important domains of general health
(fatigue, sleep disturbance, and pain interference), that are
known to be impaired in CRS. We did not find a strong cor-
relation between the change in CRS-PRO or the SNOT-22
with the PROMIS SPDSA, potentially due to the already
high level of baseline satisfaction with social activities
among our study subjects. Of the different CRS-PRO fac-
tors, we found the Rhino-Psychologic factor to correlate
best to the PROMIS Fatigue and Sleep Disturbance mea-
sures whereas the Facial Discomfort factor correlated best
with the PROMIS Pain Interference scale. Together, these
findings suggest that the CRS-PRO can meaningfully mea-
sure CRS-specific symptoms as well as impairments in
general health domains.
Unlike our prior study of patients receiving medical
therapy for CRS, the CRS-PRO and its factors did not
correlate well with objective endoscopic and CT mea-
sures in this patient population.16 Multiple prior studies
have similarly not been able to find a strong correlation
between PROMs and objective findings such as a CT scan
in presurgical CRS patients.36–38 A possible explanation
for the differences seen in our current study are that our
prior study correlated changes in the objective measures
and the changes in the CRS-PRO in the same patients
thus accounting for the variability of individual symp-
tom reporting. In the current study, only cross-sectional
presurgical CT and endoscopic scores were available, and
these were temporally separated from the PROM collec-
tion. Nonetheless, our data may suggest that of the objec-
tive findings of CRS, the CRS-PRO Rhino-Psychologic
factor may be best correlated with overall endoscopic
severity, whereas the cough factor may be best corre-
CRS-PRO responsiveness and validity in ESS
lated with the presence of middle meatal discharge. We
are unsure whether the inverse correlation between endo-
scopic edema scores and facial discomfort is a robust one
but will validate this in future studies.
We provide first evidence that the CRS-PRO total was
very responsive to the effects of ESS with very large effect
sizes noted stably at 3 and 6 months following surgery. This
responsiveness was noted in patients with CRSwNP and
CRSsNP with both groups experiencing very large reduc-
tions in their impairment. The responsiveness of the CRS-
PRO total to ESS was observed to be at least as large as those
observed with the SNOT-22. Compared to the changes
we observed in the study by Ghadersohi et al.,15,16 larger
effect sizes were noted following ESS than those with med-
ical therapy (ESS ES: 1.44 vs. medical therapy ES: 0.95).
This is consistent with the well-described overall quality
of life improvement noted by patients following ESS and
prior studies demonstrating that ESS generally results in
larger symptomatic improvements than nonbiologic med-
ical therapy for CRS.2,39 The responsiveness of the CRS-
PRO total was consistently larger than any of its individual
factors suggesting the total score was better able to holis-
tically capture improvement in the various facets of CRS
symptoms. In contrast, the SNOT-22 Rhinological subdo-
main was consistently more responsive than the SNOT-
22 total score suggesting that much of the responsiveness
of the SNOT-22 derives from this domain. This domain
of the SNOT-22 also contains multiple items targeting the
same concepts (i.e., “need to blow nose,” “thick nasal dis-
charge,” and “runny nose”) and uses items with compound
descriptors (e.g., blockage/congestion of nose) potentially
inflating the subdomain effect size.15 We further note the
CRS-PRO is also just over half the length of the SNOT-22,
and is therefore a more convenient instrument for patients,
researchers, and healthcare providers. It also has a shorter
time frame of symptom assessment than the SNOT-22 (1 vs.
2 weeks, respectively),11 and thus may be more appropriate
for measuring acute exacerbations and symptom changes.
The comparative brevity of the CRS-PRO may originate
from its conceptual framework developed after extensive
input from CRS patients. In item selection, we prioritized
elimination of highly duplicative items that were similar
either conceptually or metrically and items not reported
at significant burden to the CRS population.15 Similar to
prior studies,2 we found that although the CRS-PRO cap-
tures both disease-specific and general health items impor-
tant to CRS patients, it is more responsive than a general
health measure such as the PROMIS domains.
Finally, we also examined several methods of determin-
ing the meaningful change of the CRS-PRO using both
distribution-based and anchor-based approaches. These
estimates were relatively convergent and we propose a CID
of 6.0 because it represents the midpoint. Interestingly,
LIN et al.
we note this estimate of a CID of 6.0 was similar to an
anchor-based estimate obtained in the study by Ghader-
sohi et al.15,16 but was not expounded upon due to smaller
sample size in the study.
Strengths of this study include longer follow-up periods
of 3 months and 6 months. We had a large enough sample
size to allow for exploratory factor analysis and calculation
of CID. Limitations of this study include variable extent
of ESS, as well as the lack of CT and endoscopy scores at
follow-up and insufficient power to perform a confirma-
tory factor analysis. Furthermore, symptoms and need for
revision surgery can evolve over years after sinus surgery,
and it would be useful to investigate the CRS-PRO over
even longer timeframes. Future research should also fur-
ther assess the correlation of the CRS-PRO with objective
measures of sinonasal disease.
CONCLUSION
The CRS-PRO, developed in concordance with FDA guide-
lines for PROMs acceptable for use as end points in clini-
cal trials, is responsive to ESS and correlates strongly with
previously validated measures of CRS. Compared to the
SNOT-22, it is about half the length, and is at least equiv-
alently responsive for measuring changes before and after
ESS.
Potential conflict of interest
B.K.T. has received consultation fees from Sanofi-
Regeneron for work unrelated to this publication. S.S.S.
has received research funding from the National Institute
of Allergy and Infectious Diseases - Chronic Rhinosinusitis
Integrative Studies Program 2 (1P01AI145818-01).
Funding information
National Institutes of Health Grants: P01 AI145818-Chronic
Rhinosinusitis Integrative Studies Program (CRISP2),
R01 AI134952, R01DC016645; and the Ernest S. Bazley
Foundation.
ORCID
Katherine A. Lin BS https://orcid.org/0000-0003-4159-
7192
Caroline P.E. Price BA https://orcid.org/0000-0002-
9590-7419
Saied Ghadersohi MD https://orcid.org/0000-0002-
9742-5333
David Cella PhD https://orcid.org/0000-0002-9881-4541
1319
Robert C. Kern MD https://orcid.org/0000-0002-8995-
9175
David B. Conley MD https://orcid.org/0000-0002-2611-
6795
Stephanie Shintani-Smith MD, MS https://orcid.org/
0000-0002-0605-3993
Kevin C. Welch MD https://orcid.org/0000-0003-4863-
7206
Bruce K. Tan MD, MS https://orcid.org/0000-0001-9210-
5050
REFERENCES
1. Benninger MS, Senior BA. The development of the Rhinos-
inusitis Disability Index. Arch Otolaryngol Head Neck Surg.
1997;123:1175-1179.
2. Lidder AK, Detwiller KY, Price CPE, et al. Evaluating metrics
of responsiveness using patient-reported outcome measures in
chronic rhinosinusitis. Int Forum Allergy Rhinol. 2017;7(2):128-
134.
3. Tan BK, Klingler AI, Poposki JA, et al. Heterogeneous inflamma-
tory patterns in chronic rhinosinusitis without nasal polyps in
Chicago, Illinois. J Allergy Clin Immunol. 2017;139(2):699-703.e7.
4. Bachert C, Han JK, Desrosiers M, et al. Efficacy and safety
of dupilumab in patients with severe chronic rhinosinusitis
with nasal polyps (LIBERTY NP SINUS-24 and LIBERTY NP
SINUS-52): results from two multicentre, randomised, double-
blind, placebo-controlled, parallel-group phase 3 trials. Lancet.
2019;394(2):1638-1650.
5. Gevaert P, Omachi TA, Corren J, et al. Efficacy and safety of
omalizumab in nasal polyposis: 2 randomized phase 3 trials. J
Allergy Clin Immunol. 2020;146(3):595-605.
6. Cook KF, Jensen SE, Schalet BD, et al. PROMIS measures of
pain, fatigue, negative affect, physical function, and social func-
tion demonstrate clinical validity across a range of chronic con-
ditions. J Clin Epidemiol. 2016;73:89-102.
7. Cella D, Riley W, Stone A, et al. The Patient-Reported Out-
comes Measurement Information System (PROMIS) developed
and tested its first wave of adult self-reported health outcome
item banks: 2005-2008. J Clin Epidemiol. 2010;63:1179-1194.
8. Rudmik L, Hopkins C, Peters A, Smith TL, Schlosser RJ, Soler
ZM. Patient-reported outcome measures for adult chronic rhi-
nosinusitis: a systematic review and quality assessment. J Allergy
Clin Immunol. 2015;136:1532-1540.e2.
9. Piccirillo JF, Edwards D, Haiduk A, Yonan C, Thawley SE. Psy-
chometric and clinimetric validity of the 31-item Rhinosinusitis
Outcome Measure (RSOM-31). Am J Rhinol. 1995;9(6):297-308.
10. Piccirillo JF, Merritt MG Jr, Richards ML. Psychometric and
clinimetric validity of the 20-item Sino-Nasal Outcome Test
(SNOT-20). Otolaryngol Head and Neck Surg. 2002;126(1):41-47.
11. Hopkins C, Gillett S, Slack R, Lund VJ, Browne JP. Psychometric
validity of the 22-item Sinonasal Outcome Test. Clin Otolaryngol.
2009;34(5):447-454.
12. U.S. Department of Health and Human Services, Food and Drug
Administration. Center for Drug Evaluation and Research:
Clinical Outcome Assessment (COA) Compendium. Pub-
lished August 2019. https://www.fda.gov/drugs/development-
resources/clinical-outcome-assessment-compendium.
Accessed February 21, 2021.
1320
13. U.S. Department of Health and Human Services, Food and Drug
Administration. Guidance for Industry: Patient-Reported Out-
come Measures: Use in Medical Product Development to Sup-
port Labeling Claims. Published December 2009. https://www.
fda.gov/media/77832/download. Accessed February 21, 2021.
14. U.S. Department of Health and Human Services, Food and Drug
Administration. Principles for Selecting, Developing, Modify-
ing, and Adapting Patient Reported Outcome Instruments for
Use in Medical Device Evaluation: Draft Guidance for Industry
and Food and Drug Administration Staff, and Other Stakehold-
ers. Published Aug 2020. https://www.fda.gov/media/141565/
download. Accessed February 21, 2021.
15. Ghadersohi S, Price CPE, Jensen SE, et al. Development and pre-
liminary validation of a new patient-reported outcome measure
for chronic rhinosinusitis (CRS-PRO). J Allergy Clin Immunol
Pract. 2020;8(20):2341-2350.e1.
16. Ghadersohi S, Price CPE, Beaumont JL, et al. Responsiveness
and convergent validity of a new patient-reported outcome
measure for chronic rhinosinusitis (CRS-PRO). J Allergy Clin
Immunol Pract. 2020;8(20):2351-2359.e2.
17. Dejaco D, Riedl D, Huber A, et al. The SNOT-22 factorial struc-
ture in European patients with chronic rhinosinusitis: new clin-
ical insights. Eur Arch Otorhinolaryngol. 2019;276(5):1355-1365.
18. Feng AL, Wesely NC, Hoehle LP, et al. A validated model for
the 22-item Sino-Nasal Outcome Test subdomain structure in
chronic rhinosinusitis. Int Forum Allergy Rhinol. 2017;7(12):1140-
1148.
19. Sedaghat AR, Gray ST, Caradonna SD, Caradonna DS. Clus-
tering of chronic rhinosinusitis symptomatology reveals novel
associations with objective clinical and demographic character-
istics. Am J Rhinol Allergy. 2015;29(2):100-105.
20. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde
JG. Research electronic data capture (REDCap) – A metadata-
driven methodology and workflow process for providing
translational research informatics support. J Biomed Inform.
2009;42(2):377-381.
21. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium:
building an international community of software platform part-
ners. J Biomed Inform. 2019;95:103208.
22. Hopkins C, Slack R, Lund V, Brown P, Copley L, Browne J. Long-
term outcomes from the English national comparative audit of
surgery for nasal polyposis and chronic rhinosinusitis. Laryngo-
scope. 2009;119:2459-2465.
23. Okushi T, Nakayama T, Morimoto S, et al. A modified Lund-
Mackay system for radiological evaluation of chronic rhinosi-
nusitis. Auris Nasus Larynx. 2013;40:548-553.
24. Psaltis A, Li G, Vaezeafshar R, Cho KS, Hwang PH. Modification
of the Lund-Kennedy endoscopic scoring system improves its
reliability and correlation with patient-reported outcome mea-
sures. Laryngoscope. 2014;124:2216-2223.
25. Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Mul-
tivariate Data Analysis. New Jersey: Pearson University Press;
2006.
26. Matsunaga M. How to factor-analyze your data right: do’s,
don’ts, and how-to’s. Int J Psychol Res. 2010;3(1):97-110.
CRS-PRO responsiveness and validity in ESS
27. Akoglu H. User’s guide to correlation coefficients. Turk J Emerg
Med. 2018;18(3):91-93.
28. Sawilowsky SS. New effect size rules of thumb. J Mod Appl Stat
Methods. 2009;8(2):597-599.
29. Chen P, Lin K, Ling R, et al. Validity, responsiveness, and
minimal clinically important difference of EQ-5D-5L in stroke
patients undergoing rehabilitation. Qual Life Res. 2016;25:1585-
1596.
30. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes
in health-related quality of life: the remarkable universality of
half a standard deviation. Med Care. 2003;41(5):582-592.
31. Yost KJ, Eton DT, Garcia SF, Cella D. Minimally important dif-
ferences were estimated for six patient-reported outcomes mea-
surement information system-cancer scales in advanced-stage
cancer patients. J Clin Epidemiol. 2011;64(5):507-516.
32. Gregurić T, Trkulja V, Baudoin T, Grgić MV, Šmigovec I, Kalog-
jera L. Association between computed tomography findings and
clinical symptoms in chronic rhinosinusitis with and without
nasal polyps. Eur Arch Otorhinolaryngol. 2017;274(5):2165-2173.
33. Hopkins C, Browne JP, Slack R, et al. The national comparative
audit of surgery for nasal polyposis and chronic rhinosinusitis.
Clin Otolaryngol. 2006;31(5):390-398.
34. Lind H, Joergensen G, Lange B, Svendstrup F, Kjeldsen AD. Effi-
cacy of ESS in chronic rhinosinusitis with and without nasal
polyposis: a Danish cohort study. Eur Arch Otorhinolaryngol.
2016;273(4):911-919.
35. Streiner DL. Figuring out factors: the use and misuse of factor
analysis. Can J Psychiatry. 1994;39(3):135-140.
36. Ryan WR, Ramachandra T, Hwang PH. Correlations between
symptoms, nasal endoscopy, and in-office computed tomogra-
phy in post-surgical chronic rhinosinusitis patients. Laryngo-
scope. 2011;121:674-678.
37. Wabnitz DA, Nair S, Wormald PJ. Correlation between preop-
erative symptom scores, quality-of-life questionnaires, and stag-
ing with computed tomography in patients with chronic rhinos-
inusitis. Am J Rhinol. 2005;19:91-96.
38. Zheng Y, Zhao Y, Lv D, et al. Correlation between computed
tomography staging and quality of life instruments in patients
with chronic rhinosinusitis. Am J Rhinol Allergy. 2010;24:e41-
e45.
39. Smith TL, Kern R, Palmer JN, et al. Medical therapy vs
surgery for chronic rhinosinusitis: a prospective, multi-
institutional study with 1-year follow-up. Int Forum Allergy
Rhinol. 2013;3(1):4-9.
How to cite this article: Lin KA, Price CPE,
Huang JH, et al. Responsiveness and convergent
validity of the chronic rhinosinusitis
patient-reported outcome (CRS-PRO) measure in
CRS patients undergoing endoscopic sinus surgery.
Int Forum Allergy Rhinol. 2021;11:1308–1320.
https://doi.org/10.1002/alr.22782