Schizophrenia Research 165 (2015) 116–122

Contents lists available at ScienceDirect

Schizophrenia Research

journal homepage: www.elsevier.com/locate/schres

Longitudinal validation of psychosis risk screening tools

Emily Kline a, Elizabeth Thompson b, Caroline Demro b, Kristin Bussell c, Gloria Reeves c, Jason Schiffman b,⁎
a
Massachusetts Mental Health Center, Public Psychiatry Division of Beth Israel Deaconess Medical Center, Harvard Medical School, 75 Fenwood Road, Boston, MA 02115, USA
b
University of Maryland, Baltimore County, Department of Psychology, 1000 Hilltop Circle, Baltimore, MD 21250, USA
c
University of Maryland, Baltimore, 701 W. Pratt Street, Baltimore, MD 21201, USA

a r t i c l e i n f o a b s t r a c t

Article history: The development of widely used interview tools has helped to standardize the criteria for a “clinical high risk”
Received 9 February 2015 syndrome, thus enabling advances in efforts to develop interventions for this phase of illness. These assessments,
Received in revised form 20 April 2015 however, are burdensome to administer and not likely to be adopted for widespread use. Scalable early intervention
Accepted 23 April 2015
depends on the availability of brief, low-cost assessment tools that can serve to screen populations of interest or
Available online 11 May 2015
triage treatment-seekers toward specialized care. The current study examines the sensitivity, specificity, and
Keywords:
predictive strength of three self-report measures (Prime Screen—Revised, Prodromal Questionnaire—Brief, and
Attenuated psychosis syndrome Youth Psychosis at Risk Questionnaire—Brief) with regard to psychosis onset and symptom persistence over six
Clinical high-risk months of follow-up within an indicated sample of 54 adolescents and young adults ages 12–22. Within this
Prodrome sample, all three measures demonstrated excellent sensitivity to emerging psychosis and strong agreement with
Schizophrenia clinician evaluations of attenuated psychosis symptoms. Additionally, all screeners obtained negative predictive
Screening values of 1.00 with regard to psychosis onset, indicating that an individual scoring below the recommended
Assessment threshold score would be extremely unlikely to develop psychosis over the following six months. The longitudinal
validation of psychosis risk screening tools constitutes an important step toward establishing a standard of care for
early identification and monitoring in this vulnerable population.
© 2015 Elsevier B.V. All rights reserved.

1. Introduction Assessment of At-risk Mental States (CAARMS) in programs of CHR re-

Efforts to identify and treat psychosis during the early stages of illness
progression have stemmed from research indicating that prompt inter-
vention may maximize treatment effectiveness and quality of life
(Fusar-Poli et al., 2013). Research suggesting the relative benefits of a
short duration of untreated psychosis (Marshall et al., 2005), as well as
findings that many individuals at risk for psychosis can be clinically iden-
tified months or years before the onset of schizophrenia or other spec-
trum disorders (e.g., Cannon et al., 2008), has led to questions regarding
the possibility of identifying and treating illness during a pre-psychotic
or ‘prodromal’ phase. Effective treatment prior to the emergence of full-
blown illness holds potential to delay or even prevent the onset of acute
psychosis (Marshall and Rathbone, 2011).
Although psychosis prevention strategies are still emerging, there
have been considerable advances in screening/identification of individ-
uals designated as “clinical high risk” (CHR) for psychotic disorders. In-
terview and self-report measures have been developed to facilitate
screening among high-risk populations. In particular, the extensive
adoption of structured clinical assessments such as the Structure Inter-
view for Psychosis-risk Syndromes (SIPS) and the Comprehensive
⁎ Corresponding author at: Department of Psychology, UMBC, 1000 Hilltop Circle,
Baltimore, MD 21250, USA. Tel.: +1 410 455 1535; fax: +1 410 455 1055.
E-mail address: schiffma@umbc.edu (J. Schiffman).
search has been useful in that the common criteria are now widely rec-
ognized by researchers embarking on related but unique programs of
research (Fusar-Poli et al., 2013). Due to high clinician burden associat-
ed with training and administration, however, these measures are not
well suited for use in generalist mental health settings. The develop-
ment of brief instruments that can be ‘exported’ for clinical use is a cru-
cial step toward establishing and disseminating evidence-based care for
individuals most vulnerable to psychosis.
Brief self-report questionnaires have the potential to screen popula-
tions of interest, and may ultimately aid in the detection of far more
CHR youth than would be possible through referrals to specialized pro-
grams. Research within CHR populations suggests that these patients ex-
perience considerable distress as well as impairments in social and
occupational functioning well before the onset of full-threshold psychotic
symptoms (Addington et al., 2008; Ruhrmann et al., 2008; Velthorst et al.,
2010; Cornblatt et al., 2011; Thompson et al., 2015). Screening may be
particularly useful for identifying CHR patients at an earlier stage of illness
by detecting new-onset symptoms after they have become bothersome
to patients, but before they have caused notable impairments.
Screeners may also constitute a useful tool for specialty centers
looking to recruit CHR samples for both observational and intervention-
al research. Such programs typically administer the SIPS or CAARMS to
determine CHR status and thus eligibility for study participation;
conducting many such interviews can be a resource-intensive process.

http://dx.doi.org/10.1016/j.schres.2015.04.026
0920-9964/© 2015 Elsevier B.V. All rights reserved.
 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122 117

Table 1
CHR screening tools.

Measure Description

Prime Screen—Revised
(Miller et al., 2004)
Prodromal Questionnaire,
Brief Version (PQ-B)
(Loewy et al., 2011)
Youth Psychosis at Risk
Questionnaire (YPARQ-B)
(Ord et al., 2004)
Developed by the SIPS author group, the Prime Screen contains 12 Likert-type items describing attenuated symptoms and asks respondents to
choose from 7 response choices ranging from “definitely disagree” to “definitely agree.” In a validation sample of 36 participants, the screening
instrument showed a sensitivity of 0.90 and perfect specificity with regard to SIPS-obtained diagnoses. The authors recommend using a threshold
of two or more ‘somewhat agree’ item endorsements to categorize positive vs. negative responders. Items scored as five (somewhat agree) or six
(definitely agree) were counted as a “yes,” and the screener total was obtained by counting how many items the respondent endorsed by circling
five or six.
Average administration time: 1:40
Flesch–Kincaid reading grade level estimate: 6.8
The PQ-B is a 21-item self-report questionnaire. Items are answered true/false and also contain ‘distress scores’ in which respondents rate their
level of concern with regard to each item on a scale of 1 (‘strongly disagree’ or no distress) to 5 (‘strongly agree’ or substantial distress). Total
weighted distress scores are calculated by summing the distress scores for items that are marked ‘true.’ Respondents are asked to consider only
experiences occurring not under the influence of drugs or alcohol. Within a sample of 141 adolescents and young adults referred for assessment
to prodromal research centers, the instrument achieved specificity of 0.68 and sensitivity of 0.88 with regard to SIPS diagnoses using a cutoff
distress score threshold of six or greater on the weighted distress measure.
Average administration time: 2:30
Flesch–Kincaid reading grade level estimate: 8.6
The abbreviated YPARQ (YPARQ-B) is a 28-item self-report measure developed from the Comprehensive Assessment of At-risk Mental States.
Responses are ‘yes’ (scored as one) and ‘no/unsure’ (scored as zero). The YPARQ-B authors recommend a cut-off score of eleven endorsements
to categorize positive vs. negative responders, which yielded sensitivity of 0.82 and specificity of 0.99 in a validation sample of 648 Palauan high
school students.
Average administration time: 3:55
Flesch–Kincaid reading grade level estimate: 7.6

Screening efforts might produce an enriched sample with respect to
psychosis risk, thus optimizing assessors' time spent evaluating poten-
tially CHR individuals and limiting time spent on individuals not likely
at risk.
A handful of brief self-report measures have been developed with
the aim of assessing symptoms putatively indicating a ‘psychosis risk
state’ (Addington et al., 2014; Kline and Schiffman, 2014). Given the
prospective focus of the CHR concept, longitudinal validation represents
a vital step toward understanding how such tools might be appropriate-
ly incorporated in clinical and research settings. Although the concur-
rent agreement of CHR screening tools with clinician assessments has
been established, few studies to date have investigated longitudinal
outcomes following CHR self-report assessments (Kobayashi et al.,
2008; Loewy et al., 2012; Rietdijk et al., 2012). The goal of the current
study is to evaluate the predictive validity of three brief CHR self-
report questionnaires with regard to symptom progression and psycho-
sis onset over approximately six months within a sample of adolescents
and young adults seeking mental health treatment.
were encouraged to complete the measures on their own, but study
staff assisted teens with attention and/or literacy issues by reading
items aloud. Participants were then administered the SIPS by study
staff. Staff administering the SIPS either completed a two-day training
with the SIPS authors or were subsequently trained through a process
of reviewing vignettes provided by the SIPS authors (McGlashan et al.,
2010), practice ratings of taped interviews, observation of at least two in-
terviews, and leading at least two interviews whose ratings were com-
pared with those of an experienced interviewer. Cases were reviewed
weekly in team meetings. As an additional check on reliability, ten
audio-recorded interviews were randomly selected for re-review by the
assessment team. Inter-rater agreement for these ten interviews was
high, with intraclass correlations of 0.82 for positive symptom ratings,
0.84 for all symptom ratings, and kappa of 1.00 for diagnosis.
After six months, the participants were invited to complete a follow-
up visit at which the SIPS was re-administered.
2.2. Participants

2. Methods
2.1. Procedures
Procedures were reviewed and approved by the Institutional Review
Boards at the University of Maryland (UM) School of Medicine and Uni-
versity of Maryland, Baltimore County (UMBC). After providing informed
consent, participants completed the three self-report tools. Those under
age 18 gave assent and were required to have a parent or other legal
guardian present to provide consent. Screeners were presented in a
Latin Square design to enable detection of order effects. Participants
Participation was open to any teen or young adult ages 12–22 receiv-
ing mental health services. Participants meeting the SIPS criteria for psy-
chosis at intake were not included in the current sample. The majority
(~75%) of participants were referred by mental health clinicians both
within and outside of the UM system; others (~ 25%) self-referred
after seeing a flier posted at the UMBC campus or UM outpatient psychi-
atry clinic. Some, but not all, clinician referrals to the study were
intended as psychosis consults, thus providing a sample with potential-
ly heightened CHR prevalence.
The analysis sample includes 54 participants who completed base-
line (T1) and follow-up (T2) assessments. The 54 participants included

Table 2 Table 3
Psychosis risk diagnoses at baseline and follow-up. Descriptive statistics for primary variables.

T1 SIPS diagnosis T2 SIPS diagnoses Measure N Range Mean (SD) Skew Kurtosis Cronbach's α

Low-risk (n = 33) • 28 (stable) LR (85%) PS-R (T1) 54 0–8 2.02 (2.42) 1.10 0.17 0.92
• 5 (new-onset) CHR (15%) PQ-B (T1) 52 0–80 25.31 (23.47) 0.77 −0.54 0.95
Clinical high risk (n = 21) • 4 (remitted) LR (19%) YPARQ-B (T1) 54 0–20 7.78 (6.20) 0.41 −1.08 0.89
• 13 (persistent) CHR (62%) P-SOPS (T1) 54 0–20 8.33 (5.63) 0.30 −1.04 0.78
• 4 (transitioned) psychosis (19%) P-SOPS (T2) 54 0–22 7.22 (6.26) 0.82 −0.37 0.84
118 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122

1 risk (Miller et al., 2003). The SIPS subscales include positive, negative,
0.9
disorganization, and general symptoms, with positive symptoms figur-
ing most heavily into the conceptualization of risk syndromes. The SIPS
0.8 identifies four broad categories: Present Psychosis, Psychosis Risk Syn-
dromes, Schizotypal Personality Disorder, and no evidence of psychosis
0.7
or psychosis risk. The largest study to date to use the SIPS found that 35%
0.6 of those meeting the criteria for a SIPS psychosis risk syndrome
Sensitivity

transitioned to psychosis within 2.5 years (Cannon et al., 2008); a

0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
1 - Specificity
Fig. 1. ROC curve predicting T2 psychosis.
in the current study were drawn from an initial pool of 85 participants
who completed T1 evaluations and did not meet the SIPS criteria for a
psychotic disorder on the basis of their T1 SIPS. Of these 85, 56 (66%)
completed T2 assessments. Completers did not differ significantly
from non-completers with regard to age (t [83] = 0.68, p = .50), sex
(χ2 [1,85] = 1.18, p = .28), or baseline SIPS diagnosis (i.e., low-risk vs.
high-risk; χ2 [1,85] = 0.12, p = .73). Two participants who completed
T1 and T2 assessments were detected as likely “random responders”
and dropped from the sample, leaving a total of 54 for current
analyses. The mean duration between T1 and T2 assessments was
199.30 days (~6.5 months; SD = 23.79 days).
The sample was mostly female (n = 35, or 65%) and racially diverse
(52% African American, 31% white/European American, 13% more than
one racial category or “other,” and 4% American Indian). Mean age at in-
take was 16.20 years (SD = 3.07). The majority of participants (52%) re-
ported a family income below $40,000 annually. Eleven participants
(20%) were recruited from the UMBC campus; the remaining 80%
were recruited or referred from community clinics.
2.3. Materials
Measures used in the current study included a demographic form;
the Prime Screen—Revised (Miller et al., 2004); the Youth Psychosis At-
risk Questionnaire—Brief (YPARQ-B; Ord et al., 2004); the Prodromal
Questionnaire—Brief Version (PQ-B; Loewy et al., 2011); and the Struc-
tured Interview for Psychosis Risk Syndromes (SIPS; Miller et al.,
2003). Information about CHR screening tools is provided in Table 1
(from Kline et al., 2012).
The SIPS is a semi-structured interview targeting interviewees' ex-
periences of attenuated symptoms and other indicators of psychosis
meta-analysis found that across studies, 28% of participants meeting
PS-R the SIPS-defined CHR criteria developed psychosis (Fusar-Poli et al.,
PQ-B
2012). The positive symptom subscale of the Scale of Psychosis Risk
Symptoms (P-SOPS) was also used as a criterion measure of attenuated
YPARQ-B
symptoms. This subscale comprises clinician ratings reflecting five pos-
itive symptom domains.
2.4. Statistical analyses
0.8 0.9 1
2.4.1. Data scoring, missing data, and preliminary analyses
For categorical analyses, participants were grouped based on screener
scores and SIPS diagnoses. Participants scoring above or below the
author-recommended screening thresholds on each T1 self-report mea-
sure were categorized as ‘positive’ or ‘negative’ screens for that measure.
Participants were categorized as positive or negative ‘cases’ based on the
result of their SIPS interviews. SIPS outcome categories include current
psychosis, clinical high-risk (CHR; i.e., those meeting the criteria for one
of the psychosis risk syndromes or current schizotypal personality disor-
ders [see Addington et al., 2012]), and low-risk (LR; i.e., not meeting the
criteria for any SIPS-based diagnosis). CHR was considered to have
“persisted” from T1 to T2 for individuals who continued to meet any
SIPS-identified psychosis risk syndrome over time, even if their symp-
toms worsened or improved slightly over the follow-up period. Positive
cases were considered to have “remitted” if the individual no longer
met the SIPS CHR criteria for any risk syndrome, regardless of the
presence or absence of other psychiatric or functional difficulties.
Data were screened for outliers through examination of normality
and visual plotting of key variables (screener and P-SOPS scores).
Through this process, two participants emerged as likely ‘random re-
sponders’ and were dropped from the analysis sample. Potential order-
ing effects were examined by comparing means for scales administered
first, second, and third in the screening battery via a series of ANOVAs.
No clinician-scored data was missing. One percent of self-report data
was missing. To ensure a conservative approach to estimating screening
thresholds, missing cell values were treated as “zeros” for most vari-
ables. The exception to this rule was for PQ-B distress questions; if a par-
ticipant endorsed an item, a missing response to the “distress” question
was treated as “neutral” and scored as three (rather than zero). Some
respondents missed either a page of a measure or a measure entirely,
and in these cases the respondent was dropped for analyses using that
measure.
2.4.2. Analyses for study aims
Screeners' ability to predict T2 diagnoses was examined through a
series of 2 × 2 tables comparing T1 screen results on each instrument

Table 4
Predicting T2 psychosis from dichotomized T1 screening.

Measure AUC Threshold Sens. Spec. PPV NPV Accuracy

PS-R 0.78 (p = .06) Author recommended: ≥2 1.00 0.58 0.16 1.00 61%
Sample optimized: ≥3 0.75 0.70 0.17 0.97 70%
PQ-B 0.91 (p b .01) Author recommended: ≥6 1.00 0.33 0.11 1.00 38%
Sample optimized: ≥44 1.00 0.83 0.33 1.00 85%
YPARQ-B 0.91 (p b .01) Author recommended: ≥11 1.00 0.74 0.24 1.00 76%
Sample optimized: ≥13 1.00 0.80 0.29 1.00 81%
 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122 119

1 Table 6
Predicting T1 CHR from dichotomized T1 screening.
0.9
Measure (threshold) Outcome Sens. Spec. PPV NPV Accuracy
0.8
T1 PS-R (≥2) SIPS T1 0.71 0.70 0.60 0.79 70%
0.7 T1 PQ-B (≥6) SIPS T1 0.95 0.47 0.53 0.94 65%
T1 YPARQ-B (≥11) SIPS T1 0.62 0.88 0.76 0.78 78%
0.6
Sensitivity

0.5

0.4
0.3
0.2
0.1
0
0 0.2 0.4 0.6
1 - Specificity
Fig. 2. ROC curve predicting T2 SIPS status.
to T2 diagnoses. Sensitivity, specificity, positive predictive value, and
negative predictive value were calculated for each screener with regard
to outcomes of interest. Screening scores were entered into Receiving
Operator Characteristic (ROC) curves differentiating participants based
on the state variable of interest (psychosis vs. no psychotic disorder;
or CHR/psychosis vs. no SIPS diagnosis). Area Under the Curve (AUC)
was examined as a measure of screeners' accuracy with regard to diag-
nostic status. The minimum distance between each coordinate on the
ROC curve and the point (0,1) was identified in order to identify the op-
timal cutpoint on each measure (Kumar and Indrayan, 2011).
To examine the screeners' prediction of symptom severity over time,
correlations between each T1 screener and T2 P-SOPS ratings were calcu-
lated. To provide a sense of relative stability, correlations between T1
screeners and T1 P-SOPS ratings were also calculated. Correlations be-
tween T1 self-report measures and T2 clinician ratings were compared
via R to Z tests to determine whether differences between correlation
magnitudes were statistically significant (Dunn and Clark, 1969;
Steiger, 1980).
PS-R
the area under the curve (AUC), sensitivity, specificity, positive and neg-
PQ-B
ative predictive values, and overall accuracy of each screener (relative to
YPARQ-B
psychosis outcomes) when scores were dichotomized at both author-
recommended and ROC-suggested thresholds.
Given the relatively short duration of follow-up, it is relevant to as-
sess the screeners' sensitivity to both psychosis and CHR outcomes,
0.8 1 since participants with persistent CHR symptoms may still be likely to
progress toward psychosis over time. To this end, analyses were repeat-
ed with outcomes reconceptualized as CHR/psychosis vs. no diagnosis at
T2. Fig. 2 displays the ROC curves obtained using this alternative out-
come strategy. Table 5 displays the area under the curve (AUC), sensitiv-
ity, specificity, positive and negative predictive values, and overall
accuracy of each screener when predicting CHR/psychosis vs. no diag-
nosis when scores were dichotomized at both author-recommended
and ROC-suggested thresholds.
In order to compare screeners' accuracy when predicting T1
(concurrent) vs. T2 (future) CHR/psychosis diagnoses, we also exam-
ined the T1 screeners' accuracy when predicting concurrent (T1) SIPS
diagnoses using author-recommended thresholds (Table 6). Fig. 3 pro-
vides a visual representation of cases identified by each measure
(screeners and SIPS) at T1, with cases that transitioned to psychosis by
T2 represented by stars.
Finally, symptom severity was examined from a continuous per-
spective using P-SOPS scores. Pearson correlations between T1 screen-
ing totals and T1 and T2 P-SOPS scores indicated strong associations
between self-report and clinician-administered measures of symptom
severity (Table 7).
4. Discussion

Brief self-report questionnaires have the potential to screen popula-

3. Results
SIPS diagnoses following T1 and T2 interviews, as well as trajectories
for each initial diagnostic group, are described in Table 2. Of those who
were initially LR, none transitioned to psychosis. Of the 21 participants
meeting the SIPS CHR criteria at T1, 4 (19%) had transitioned to psycho-
sis by T2. Table 3 provides descriptive statistics and sample size for anal-
yses involving each measure. Given that skew and kurtosis were less
than or equal to 1.1 for all variables, Pearson correlations were used
for analyses. ANOVAs conducted to compare means for self-report mea-
sures depending on their placement within the battery did not detect
significant differences, suggesting that the order of measures did not
substantially impact responses.
Screeners' ability to predict T2 psychosis was examined via receiver
operating characteristic (ROC) curve estimation (Fig. 1). Table 4 displays
tions of interest, and may ultimately aid in the detection of far more CHR
youth than would be possible through clinician- or self-referrals to spe-
cialized programs. Prior research also supports the use of screeners as
“triage” tools that may be helpful for determining whether youth pre-
senting for mental health services may benefit from more specialized,
psychosis-oriented treatment. One of the most comprehensive screen-
ing studies to date used the PQ to screen young adults seeking mental
health services in the Netherlands. Public psychiatric clinics adopted
the PQ as a screening tool to detect possible CHR cases among young
adults ages 18–35 upon their first contact with the mental health sys-
tem (Rietdijk et al., 2012). This method of identifying high-risk patients
via screening was compared to a referral method, in which clinicians
were asked to refer suspected high-risk cases to a specialty clinic for as-
sessment. Relative to the referral method, the screening method yielded
a greater number of identified CHR cases (147 vs. 66) within a smaller

Table 5
Predicting T2 CHR/psychosis from dichotomized T1 screening.

Measure AUC Threshold Sens. Spec. PPV NPV Accuracy

PS-R 0.79 (p b .01) Author recommended: ≥2 0.77 0.75 0.68 0.83 76%
Sample optimized (same)
PQ-B 0.83 Author recommended: ≥6 0.86 0.42 0.50 0.81 60%
(p b .01) Sample optimized: ≥29 0.81 0.87 0.81 0.87 85%
YPARQ-B 0.86 (p b .01) Author recommended: ≥11 0.64 0.91 0.82 0.78 80%
Sample optimized: ≥10 0.73 0.86 0.84 0.76 80%
120 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122
Fig. 3. Classification of cases by screeners.
population catchment area. Compared to those detected by referral, a
significantly greater number of CHR cases detected by screening con-
verted to psychosis (27% vs. 9%), suggesting that the PQ-B was better
able to detect those prodromal to psychosis relative to clinician impres-
sions (Rietdijk et al., 2012). These findings strongly suggest that screen-
ing within psychiatric clinics holds promise as an effective method for
identifying a large number of the highest-risk patients.
Findings from the current study align well with previous findings in
recommending the use of self-report tools for detecting CHR cases with-
in a young help-seeking population. Over six months of follow-up, all
three measures demonstrated excellent sensitivity with regard to psy-
chosis onset. Participants who went on to develop psychosis were
flagged by all three screeners. Further, participants scoring above the
screening thresholds on the PS-R and YPARQ-B at T1 were significantly
more likely to transition to psychosis and/or continue to experience per-
sistent attenuated symptoms over the course of follow-up. These results
suggest that screeners can be used to select an “enriched” group that is
likely to experience persistent problems over time. Applying the current
study's positive predictive values somewhat speculatively at the indi-
vidual level, findings indicate that an individual scoring above screening
cutoffs on the PS-R, PQ-B, or YPARQ-B has respectively a 60%, 53%, or
76% likelihood of meeting the SIPS CHR criteria as well. These figures
do not change substantially when outcomes are observed over six
months; the person who scored over the PS-R/PQ-B/YPARQ-B threshold
then has a 68%, 50%, or 82% likelihood of continuing to meet these
criteria over time. The negative predictive values obtained by all three
instruments were notably strong. Using author-recommended thresh-
olds, each of the measures demonstrated perfect NPV of 1.00; in other
words, an individual could be effectively “ruled out” of an imminent
risk category in our sample on the basis of a negative screen. From a clin-
ical care perspective, these findings suggest that CHR programs that
focus on conversion as an outcome should consider the results of an ini-
tial screen before administering a full SIPS or other clinical interview.
For the purpose of comparing the relative merits of the three instru-
ments' performance, a few observations are apparent. The YPARQ-B and
PS-R demonstrated reliable score thresholds that generalized fairly well
from authors' validation samples to the current population, in contrast
to the PQ-B. Although it is not clear why the PQ-B recommended cutoffs
differed so dramatically from the optimized cutoff obtained in the cur-
rent sample, it is possible that the variation could be attributed to differ-
ences in local samples, or to reliability issues within the measure itself. It
is interesting to note that a recent study by Zhang et al. (2014) also sug-
gested that the PQ-B's author recommended cut might be too low for
adequate specificity.
Used as continuous measures, however, the PQ-B and YPARQ-B cor-
related more highly with clinician-rated symptoms than the PS-R with
Table 7
Pearson correlations between self-report and P-SOPS.
P-SOPS (T1) P-SOPS (T2)
PS-R (T1) .63⁎,a .56⁎,c
PQ-B (T1) .77⁎,b .71⁎,d
YPARQ-B (T1) .76⁎,b .71⁎,d
a,b
In R to Z comparison, the PQ-B and YPARQ-B correlate significantly more highly with con-
current (T1) P-SOPS totals relative to the PS-R (PS-R vs. PQ-B: t = −2.35, df = 51, p = .023;
PS-R vs. YPARQ-B: t = −2.52, df = 51, p = .015), but did not significantly differ from one
another (PQ-B vs. YPARQ-B: t = −0.26, df = 51, p = .798).
c,d
R to Z comparisons indicated that the PQ-B and YPARQ-B correlated significantly more
highly with T2 P-SOPS totals relative to the PS-R (PS-R vs. PQ-B: t = −2.28, df = 51, p =
.027; PS-R vs. YPARQ-B: t = −2.69, df = 51, p = .010), but did not differ from one another
(PQ-B vs. YPARQ-B: t = 0).
⁎ p b .01.
 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122
regard to agreement with both concurrent and future P-SOPS scores
(Table 7). For a more comprehensive discussion of the sensitivity, spec-
ificity, and uses to date of various screening instruments (including the
three tested within the current study), see the Kline and Schiffman
(2014) or Addington et al. (2014) recent reviews of the psychosis risk
screening literature.
4.1. Limitations
This study has several limitations. First, the small sample size leads
to several potential liabilities, in particular the possibility for Type II
error and/or unreliable findings. Second, the duration of follow-up (on
average, about 6.5 months) constitutes a relatively brief window rela-
tive to larger studies that have followed CHR individuals over time.
This may have been too short to observe clinical deterioration among
some participants who may be in a truly “prodromal” state, but who
did not decompensate during the observation period. On the other
hand, six months constitute a realistic window for clinical monitoring.
Third, our findings may have been biased in some way by a less than
ideal retention rate within the sample initially seen for a T1 visit. Of
the 85 participants eligible for follow-up, 56 (66%) completed a T2
visit within the designated window of 6 to 9 months, 54 of whom
were retained in the final analysis sample. Although less than the reten-
tion rates reported by large studies of clinical high-risk participants
(e.g., 79% retention reported by Cannon et al. (2008); 74% retention re-
ported by Ruhrmann et al. (2010)), this retention rate is similar to those
reported for other, somewhat smaller CHR protocols (e.g., 59% reported
by Morita et al. (2014); 67% reported by Schlosser et al. (2012)).
Finally, the unique characteristics of the current sample may prevent
findings from generalizing to other populations. Given that several par-
ticipants were referred by clinicians concerned about psychosis, the
prevalence of CHR/psychosis was higher than expected, and may have
affected the performance of screeners (in particular, the obtained PPV
and NPV values, which unlike sensitivity and specificity, may be influ-
enced by population prevalence of the condition of interest). The sam-
ple was primarily female, which may have influenced screening and/
or interview responses. Further, our sample only included individuals
already seeking or receiving mental health services, and thus perhaps
more comfortable disclosing symptoms than youth unfamiliar with an-
swering questions about potentially sensitive symptoms.
4.2. Directions for future research
In order for screening to be maximally effective, several unanswered
questions must be addressed. The instability of optimal screening thresh-
olds across populations and settings constitutes a major barrier to suc-
cessful screening efforts. This variability may be due to setting-specific
differences in culture (e.g., the meaning attributed to psychotic-like expe-
riences), developmental norms (e.g., some items may reflect beliefs that
are more common among teens), and clinical severity of cases
(e.g., inpatient psychiatry unit vs. college counseling center). Without
sensitivity to developmental and cultural norms, it is difficult to deter-
mine what constitutes a clinically meaningful elevation. Understanding
population norms is especially important in the context of increased ex-
pectations for mental health screening in primary care in the United
States (Mechanic, 2012).
Finally, the current study cannot offer insight regarding potential
harms of psychosis risk screening. Stigma, in particular, may impact
the risk to benefit ratio of screening and early intervention efforts
(Corcoran et al., 2005). Screening could create distress or even negative-
ly affect an individual's trust in his or her providers. The likelihood of
harm may vary across settings and depend largely on the clinical and
cultural sensitivity with which a screening protocol is implemented. Pa-
tients' perceptions of psychosis risk screening and the availability of ap-
propriate resources for treatment constitute important considerations
when evaluating the potential impact of screening.
121
Role of funding source
This work was supported in part by funding from the Maryland Department of Health
and Mental Hygiene, Mental Hygiene Administration through the Center for Excellence on
Early Intervention for Serious Mental Illness (OPASS# 14-13717G/M00B4400241). The
funders had no influence on study design, analyses, interpretations, or decision to submit
manuscript for publication.
Contributors
Dr. Kline oversaw the study design, data analysis, data interpretation, and manuscript
preparation. Ms. Thompson and Ms. Demro contributed to the data collected, data analy-
ses, and manuscript preparation. Ms. Bussell oversaw the protocol implementation and
contributed to the study design. Drs. Schiffman and Reeves oversaw the protocol develop-
ment and implementation and contributed to manuscript preparation.
Conflict of interest
The authors have no actual or potential conflicts of interest to report.
Acknowledgment
The authors wish to acknowledge the generous contributions of the study's partici-
pants and their families.
References
Addington, J., Penn, D., Woods, S., Addington, D., Perkins, D., 2008. Social functioning in
individuals at clinical high risk for psychosis. Schizophr. Res. 99 (1–3), 119–124.
Addington, J., Cadenhead, K.S., Cornblatt, B.A., Mathalon, D.H., McGlashan, T.H., Perkins,
D.O., et al., 2012. North American Prodrome Longitudinal Study (NAPLS 2): overview
and recruitment. Schizophr. Res. 142 (1–3), 77–82.
Addington, J., Stowkowy, J., Weiser, M., 2014. Screening tools for clinical high risk for psy-
chosis. (online ahead of print). Early Interv. Psychiatry.
Cannon, T.D., Cadenhead, K., Cornblatt, B., Woods, S.W., Addington, J., Walker, E., et al.,
2008. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal
study in North America. JAMA Psychiatry 65 (1), 28–37.
Corcoran, C., Malaspina, D., Hercher, L., 2005. Prodromal interventions for schizophrenia
vulnerability: the risks of being “at risk”. Schizophr. Res. 73 (2), 173–184.
Cornblatt, B.A., Carrión, R.E., Addington, J., Seidman, L., Walker, E.F., Cannon, T.D., et al.,
2011. Risk factors for psychosis: impaired social and role functioning. Schizophr.
Bull. sbr136.
Dunn, O.J., Clark, V., 1969. Correlation coefficients measured on the same individuals.
J. Am. Stat. Assoc. 64 (325), 366–377.
Fusar-Poli, P., Bonoldi, I., Yung, A.R., Borgwardt, S., Kempton, M.J., Valmaggia, L., et al.,
2012. Predicting psychosis: meta-analysis of transition outcomes in individuals at
high clinical risk. JAMA Psychiatry 69 (3), 220–229.
Fusar-Poli, P., Borwardt, S., Bechdolf, A., Addington, J., Reicher-Rössler, A., Schultze-Lutter,
F., et al., 2013. The psychosis high-risk state: a comprehensive state-of-the-art re-
view. JAMA Psychiatry 70 (1), 107–120.
Kline, E., Schiffman, J., 2014. Psychosis risk screening: a systematic review. Schizophr. Res.
158 (1), 11–18.
Kline, E., Wilson, C., Ereshefsky, S., Denenny, D., Thompson, E., Pitts, S.C., et al., 2012. Psy-
chosis risk screening in youth: a validation study of three self-report measures of at-
tenuated psychosis symptoms. Schizophr. Res. 141 (1), 72–77.
Kobayashi, H., Nemoto, T., Koshikawa, H., Osono, Y., Yamazawa, R., Murakami, M., et al.,
2008. A self-reported instrument for prodromal symptoms of psychosis: testing the
clinical validity of the PRIME Screen—Revised (PS-R) in a Japanese population.
Schizophr. Res. 106 (2–3), 356–362.
Kumar, R., Indrayan, A., 2011. Receiver operating characteristic (ROC) curve for medical
researchers. Indian Pediatr. 48 (4), 277–287.
Loewy, R.L., Pearson, R., Vinogradov, S., Bearden, C.E., Cannon, T.D., 2011. Psychosis risk
screening with the Prodromal Questionnaire — Brief version (PQ-B). Schizophr. Res.
129 (1), 42–46.
Loewy, R.L., Therman, S., Manninen, M., Huttunen, M.O., Cannon, T.D., 2012. Prodromal
psychosis screening in adolescent psychiatry clinics. Early Interv. Psychiatry 6 (1),
69–75.
Marshall, M., Rathbone, J., 2011. Early intervention for psychosis. Schizophr. Bull. 37 (6),
1111–1114.
Marshall, M., Lewis, S., Lockwood, A., Drake, R., Jones, P., Croudace, T., 2005. Association
between duration of untreated psychosis and outcome in cohorts of first-episode pa-
tients. JAMA Psychiatry 62 (9), 975–983.
McGlashan, T.H., Walsh, B., Woods, S., 2010. The Psychosis Risk Syndrome: Handbook for
Diagnosis and Follow-up. Oxford University Press, New York.
Mechanic, D., 2012. Seizing opportunities under the Affordable Care Act for
transforming the mental and behavioral health system. Health Aff. 31 (2),
376–382.
Miller, T.J., McGlashan, T.H., Rosen, J.L., Cadenhead, K., Ventura, J., McFarlane, W., et al.,
2003. Prodromal assessment with the structured interview for prodromal syndromes
and the scale of prodromal symptoms: predictive validity, interrater reliability, and
training to reliability. Schizophr. Bull. 29 (4), 703–715.
Miller, T.J., Cicchetti, D., Markovich, P.J., McGlashan, T.H., Woods, S.W., 2004. The SIPS
screen: a brief self-report screen to detect the schizophrenia prodrome. Schizophr.
Res. 70 (Suppl. 1), 78.
Morita, K., Kobayashi, H., Takeshi, K., Tsujino, N., Nemoto, T., Mizuno, M., 2014. Poor out-
come associated with symptomatic deterioration among help-seeking individuals at
risk for psychosis: a naturalistic follow-up study. Early Interv. Psychiatry 8 (1), 24–31.
122 E. Kline et al. / Schizophrenia Research 165 (2015) 116–122
Ord, L., Myles-Worsley, M., Blailes, F., Ngiralmau, H., 2004. Screening for prodromal ado-
lescents in an isolated high-risk population. Schizophr. Res. 71 (2–3), 507–508.
Rietdijk, J.J., Klaassen, R.R., Ising, H.H., Draqt, S., Nieman, D.H., van de Kamp, J., et al., 2012.
Detection of people at risk of developing a first psychosis: comparison of two recruit-
ment strategies. Acta Psychiatr. Scand. 126 (1), 21–30.
Ruhrmann, S., Paruch, J., Bechdolf, A., Pukrop, R., Wagner, M., Berning, J., et al., 2008. Re-
duced subjective quality of life in persons at risk for psychosis. Acta Psychiatr. Scand.
117 (5), 357–368.
Ruhrmann, S., Schultze-Lutter, F., Salokangas, R.K.R., Heinimaa, M., Linszen, D.,
Dingemans, P., et al., 2010. Prediction of psychosis in adolescents and young adults
at high risk: results from the prospective European prediction of psychosis study.
Arch. Gen. Psychiatry 67 (3), 241–251.
Schlosser, D.A., Jacobson, S., Chen, Q., Sugar, C.A., Niendam, T.A., Li, G., et al., 2012. Recov-
ery from an at-risk state: clinical and functional outcomes of putatively prodromal
youth who do not develop psychosis. Schizophr. Bull. 38 (6), 1225–1233.
Steiger, J.H., 1980. Tests for comparing elements of a correlation matrix. Psychol. Bull. 87
(2), 245–251.
Thompson, E., Kline, E., Ellman, L., Mittal, V., Reeves, G., Schiffman, J., 2015. Emotional and
behavioral symptomatology reported by help-seeking youth at clinical high-risk for
psychosis. Schizophr. Res. 162 (1–3), 79–85.
Velthorst, E., Nieman, D.H., Linszen, D., Becker, H., de Haan, L., Dingemans, P.M., et al.,
2010. Disability in people clinically at high risk of psychosis. Br. J. Psychiatry 197
(4), 278–284.
Zhang, T., Li, H., Woodberry, K.A., Seidman, L.J., Zheng, L., Li, H., et al., 2014. Prodromal psy-
chosis detection in a counseling center population in China: an epidemiological and
clinical study. Schizophr. Res. 152 (2), 391–399.