562912

research-article2014
JPCXXX10.1177/2150131914562912Journal of Primary Care & Community HealthValleley et al

Pilot Studies
Journal of Primary Care & Community Health

Behavioral Health Screening in

2015, Vol. 6(3) 199­–204
© The Author(s) 2014
Reprints and permissions:
Pediatric Primary Care: A Pilot Study sagepub.com/journalsPermissions.nav
DOI: 10.1177/2150131914562912
jpc.sagepub.com

Rachel J. Valleley1, Natalie Romer2, Sara Kupzyk1,

Joseph H. Evans1, and Keith D. Allen1

Abstract
This pilot study investigated the effect of the Pediatric Symptom Checklist (PSC) on identification of and physician response
to behavioral health (BH) concerns. Researchers reviewed 1211 charts of youth aged 4 to 16 years. Records were
compared during baseline and an intervention consisting of implementation of the PSC to determine the rate of BH
identification and pediatrician response. Access to PSC data resulted in a trivial difference in BH concerns identified by
physicians and did not affect physician responses. This case study demonstrates that simply implementing BH screening in
primary care may not result in improved outcomes for these children.

Keywords
children, impact evaluation, pediatrics, primary care, program evaluation

In primary care (PC), 15% to 25% of children present with Method

a behavioral health (BH) concern.1,2 These problems are
managed most effectively if detected and treated in child-
hood,3 and PC has been deemed a major point of access to
brief and early intervention.4 Unfortunately, primary care
physicians (PCPs) have struggled to fill this role5 and many
children with BH concerns are not detected early or referred
for services.6
One solution to early detection is to use BH screening.
BH screening has been recommended as a cost-effective
and efficient practice for early identification of BH con-
cerns in PC.7,8 Previous research has focused on validating
screening tools that identify patients at risk for psychopa-
thology,9,10 including the Pediatric Symptom Checklist.11
Despite the support for screening instruments,12 it is not
clear how BH screening affects actual physician behavior.
Research to date on these issues has been mixed. One study
found that there was a significant increase in rates of refer-
rals when the PSC was implemented in health clinics serv-
ing a low-income, Hispanic community.13 However, other
studies have found that few children identified with BH
concerns are referred for services14,15 and that screening
alone may have little impact on physician practices in terms
of referral or initiation of treatment.16,17 Thus, there is still
much to learn about how BH screening affects physician
identification and referral of children with BH concerns.
The purpose of this case study was to examine how the
implementation of a BH screening instrument affected
PCPs’ identification and response to BH concerns in a real-
world setting.
Participants and Setting
A privately owned pediatric PC clinic in a Midwestern city
participated. Seven pediatricians (5 full time; 3 female),
with an average of 11 years of experience (range 3-17 years)
staffed this clinic. BH services were provided on-site 2 days
per week by a psychologist.
Researchers reviewed electronic charts for 1211 patient
visits with ages ranging from 4 to 16 years attending visits.
Table 1 summarizes characteristics of the visits for the 2
groups (screening and no screening).
Measure
Independent Variable. Implementation of the Pediatric
Symptom Checklist (PSC) was the independent variable.
The PSC is a 1-page, 35-item screening questionnaire
designed to capture a parent’s impression of their child’s
psychosocial functioning in a PC setting.11,18 Each item is
assigned a score of 2 = often; 1 = sometimes; and 0 = never.
Cut scores are provided for different ages. Parents are also
asked 2 additional questions: if their child has a BH concern
1
University of Nebraska Medical Center, Omaha, NE, USA
2
University of South Florida, Tampa, FL, USA
Corresponding Author:
Rachel J. Valleley, Munroe-Meyer Institute, 985450 University of
Nebraska Medical Center, Omaha, NE 68144, USA.
Email: rvallele@unmc.edu
200 Journal of Primary Care & Community Health 6(3)

Table 1.  Descriptive Characteristics of the No Screening and Screening Groups (N = 1,211).

No Screening Screening

  n % n %
Patient gender
 Female 282 42.3 262 48.2
 Male 385 57.7 282 51.8
Age, years
 4-7 302 45.3 199 36.6
 8-12 227 34.0 215 39.5
 13-16 138 20.7 130 23.9
Visit with primary care physician
 Yes 541 81.1 488 89.7
 No 126 18.9 56 10.3
Type of visit
 Sick 300 45.0 127 23.3
 Well 367 55.0 415 76.3
 Unknown  
Existing behavioral health concern
 Yes 54 8.1 53 9.7
 No 613 91.9 491 90.3
Current psychotropic medication
 Yes 43 6.4 33 6.1
 No 624 93.6 511 93.9
 Unknown 0 0.0 2 0.4
Newly identified behavioral health concerna
 Total 58 8.7 73 13.4
  Physician 1 (part time) 1 2.7 8 13.1
  Physician 2 12 7.2 13 9.5
  Physician 3 8 6.4 6 11.8
  Physician 4 12 12.5 13 13.3
  Physician 5 20 14.5 24 20.5
  Physician 6 5 5.4 7 10
  Physician 7 (part time) 0 0 2 20.0
 Average 8.3 7.0 10.4 14.0
Action taken for new behavioral health concernb
 Total 22 37.3 22 29.3
  Physician 1 (part time) 1 100 1 12.5
  Physician 2 3 18.8 3 21.4
  Physician 3 5 55.6 3 50.0
  Physician 4 4 50.0 3 21.4
  Physician 5 7 35.0 7 29.2
  Physician 6 2 40.0 3 42.9
  Physician 7 (part time) N/A N/A 2 100
 Average 3.1 50.0 3.14 39.6
a
Percentage calculated as number of newly identified behavioral health concerns divided by total number of patients seen.
b
Percentage calculated as number of actions taken for newly identified behavioral health concerns divided by total number of patients newly identified.

that he/she needs help with and if they would like additional
services. It has been found to be reliable18 and valid.18-20
Dependent Variables.  The primary dependent variables were
whether a BH concern was identified during the visit and
whether the pediatrician addressed the BH concern. Newly
identified BH concerns and physician responses were ascer-
tained from a variety sources in the visit note (eg, physician
made note of a BH concern, physician added a new BH
diagnosis, physician prescribed a psychotropic medication
for first time, noted psychology referral, counseled on
behavior).
Valleley et al 201
Procedure
A quasi-experimental research design was employed, con-
sisting of 2 groups: no screening (5 weeks) and screening
using the PSC (6 weeks). Prior to PSC implementation,
pediatricians were provided with a description of the PSC.
During PSC implementation, researchers approached care-
givers and provided them with the PSC to complete. The
researcher scored the PSC and placed it at the pediatri-
cian’s station. Pediatricians initialed PSC forms after
review. In all, 82.9% of the PSC forms scanned into patient
charts were initialed. The administration of the PSC was
managed by the research team to ensure that the majority
of patients were screened; however, decision making
regarding whether a child was at-risk for a BH concern and
management of the BH concern was left to the PCP.
Charts were reviewed by the researchers. Coded data
included whether the pediatrician identified a BH concern
and type, pediatrician response to the BH concern, whether
an existing BH concern had been identified and if services
were being provided, whether the PSC was in the at-risk
range, and whether the caregiver requested assistance or
services on the PSC.
A second rater coded data for 20.3% of the sample.
Interrater agreement was calculated as the number of
instances of agreement divided by agreements plus dis-
agreements multiplied by 100%. Percentage agreement for
the primary dependent variables was as follows: identified
BH concern = 94.7%; pediatrician addressed BH concern =
91.5%.
Results
Descriptive Analysis
All providers detected new behavioral health concerns in
their patient population. The percentages for full-time pro-
viders ranged from 5.4% to 14.5% during the no screening
phase and from 9.5% to 20.5% in the screening phase. All
pediatricians identified a higher percentage of patients with
a behavioral health concern during screening. Physician
response to the concerns varied across full time providers
ranging from 18.8% to 55.6% during no screening and
21.4% to 50% during screening.
In the no-screening group, 8.7% of visits included docu-
mentation of newly identified BH concern(s). In visits with
newly identified BH concern(s), physicians responded to
37.3%. During the screening phase, new BH concern(s)
were identified for 13.4% of the 544 patients, and of these,
pediatricians responded to 29.3%.
Based on caregiver ratings on the PSC, 5.0% of patients
(n = 27) exceeded the cut score indicating risk of a BH
problem. Of the 27 identified as at-risk, 11.1% had a BH
concern identified, 44.4% had a previous BH concern
documented, and 48.1% had existing behavioral health ser-
vices documented.
When responding to the 2 questions at the end of the
PSC, 30 caregivers, regardless if the PSC was at risk, indi-
cated that their child had a behavioral problem for which he
or she needed help. Notes showed that pediatricians
addressed six of these concerns. Twenty caregivers indi-
cated interest in services to address their child’s behavioral
problem, and of these, pediatricians documented taking
action 5 times.
Figure 1 summarizes the types of newly identified BH.
Most common problems identified were feeding, sleep, and
attention deficit/hyperactivity disorder (ADHD) symptoms.
The pediatricians were most likely to provide a behavioral
health recommendation to address concerns (Figure 2).
Statistical Analysis
Behavioral Health Concern Identified.  Results of a 2-vari-
able chi-square test showed a significant difference in the
proportion of BH concerns identified between the screen-
ing and no screening groups: χ2(N = 1211, 1) = 6.93, P =
.008. In the no-screening group, behavioral health con-
cerns were identified in 8.7% (n = 58) of visits and in the
screening group behavioral health concerns were identi-
fied in 13.4% (n = 73) of visits. When reviewing the out-
put from the chi-square test, we also made the observation
that these 2 percentages were similar, yet the chi-square
test significant. Thus, given the large sample (n = 1211),
we decided to also consider effect size, which was indeed
small (r = 0.075 and when converted Cohen’s d = 0.15).
This suggests that although the chi-square test was statisti-
cally significant, the difference was not meaningful.
Pediatrician Response to Identified Behavioral Health Con-
cern.  A 2-variable chi-square test was also used to evaluate
differences in the proportion of cases where BH concerns
where identified and the pediatrician responded between
the conditions. The difference in the proportion of patients
with identified BH concerns where the pediatrician
responded was not significantly different between groups:
χ2(N = 131, 1) = 0.88, P = .348.
Discussion
Access to BH screening data in this pediatric office was
not sufficient in increasing the number of identified BH
concerns by pediatricians; nor did implementation of the
PSC affect pediatrician response to identified BH con-
cerns. During all 1211 visits in both the screening and no
screening phases, pediatricians documented 10.8% of
patients having an identified BH concerns and 8.8% of
the patients having had a previously identified BH
202 Journal of Primary Care & Community Health 6(3)
Figure 1.  Proportion of different types of newly identified behavioral health concerns by group (no screening n = 73, screening n = 58).
concern. However, physicians responded to only 33% of
identified BH concerns. Of the patients whose caregivers
completed the PSC, pediatricians documented identified
BH concern(s) for 13.4%. Interestingly, only 5.0% of the
patients whose caregivers completed a PSC exceeded the
cut score (ie, fewer BH concerns were identified using the
PSC than practice as usual). In the majority of cases
where a child was identified as at-risk or parents indi-
cated that they were interested in seeking assistance,
pediatricians did not take action.
Pediatric settings have been identified as an optimal set-
ting for behavioral screening, yet, our findings are consis-
tent with research demonstrating that when BH concerns
are identified in PC, problems are often undertreated and
receive minimal follow-up.21,22
This study was conducted in a metro area clinic, with a
population of primarily English speaking, and privately
insured families. The clinic also has a colocated psycholo-
gist providing services. When considering this context,
our findings are particularly discouraging given that pre-
ventative care adherence has been associated with
patients being better insured, English speaking, and
nonminority.23,24
Limitations
This study has some limitations that may have affected the
results. First, implementation of the PSC by the researchers
may have limited integration of the screening procedure into
the day-to-day procedures. Second, there were no direct mea-
sures of procedural integrity. Pediatricians reviewed the
majority of the completed PSCs; however, we do not know if
pediatricians reviewed the PSC prior to, during, or following
the appointment. Our review of records focused on the notes
from the appointment during which the PSC was completed
and any prior records. Pediatricians may have followed up on
an elevated PSC score during a future encounter. Third, this
was a sample of convenience and external validity is limited
given that data were collected within 1 clinic. Finally, given
that the data were collected from patients’ charts, it is possi-
ble that physicians discussed concerns with the patient, but
failed to document information.
Valleley et al 203
Figure 2.  Proportion of types of actions taken by the pediatricians to address the newly identified behavioral health concerns by
phase (phase A no screening, phase B screening).
Future Directions
This study demonstrated that implementation of the PSC in
this clinic did not improve the identification or response by
pediatricians to identified BH concerns. This may be
explained by some of the limitations, but it may also be an
example of the gap between evidence-based practice and
implementation.25,26 Research is needed to identify the
essential, valued, feasible, and effective components of BH
screening procedures within PC and evaluate outcomes
over time. Our findings are aligned with the call for more
information and training on the requisite systems for effec-
tive screening and follow-up within PC.27,28
The psychometric validity of BH screening measures
has been well established in the research. If pediatricians
are to commit to integrating screening into PC, however,
more research is needed on implementation (eg, provider/
staff education and materials, pay-for performance initiatives
targeted at screening, automated identification such as chart
flagging, improvements in office procedures) that results in
improved access to care for children and adolescents identi-
fied at-risk, and ultimately, improved outcomes over time.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) disclosed receipt of the following financial support
for the research, authorship, and/or publication of this article: This
work was supported by grant numbers HRSA D40HP02597-08-00
and M01HP25184-01-00.
References
1. Cooper S, Valleley RJ, Polaha J, Begeny J, Evans JH.
Running out of time: physician management of behavioral
health concerns in rural pediatric primary care. Pediatrics.
2006;118:e132-e138.
2. Briggs-Gowan MJ, Horwitz SM, Schwab-Stone ME,
Leventhal JM, Leaf PJ. Mental health in pediatric settings:
distribution of disorders and factors related to service use. J
Am Acad Child Adolesc Psychiatry. 2000;39:841-849.
3. Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ,
Poulton R. Prior juvenile diagnoses in adults with mental
204 Journal of Primary Care & Community Health 6(3)
disorder: developmental follow-back of a prospective-longi-
tudinal cohort. Arch Gen Psychiatry. 2003;60:709-717.
4. Tolan PH, Dodge KA. Children’s mental health as a primary
care and concern: a system for comprehensive support and
service. Am Psychol. 2005;60:601-614.
5. Sheldrick RC, Merchant S, Perrin EC. Identification of devel-
opmental-behavioral problems in primary care: a systematic
review. Pediatrics. 2011;128:356-363.
6. Cooper JL, Masi R, Vick J. Social and Emotional Development
in Early Childhood: What Every Policymaker Should Know.
New York, NY: National Center for Children in Poverty;
2009.
7. Hogan MF. New Freedom Commission report: The
President’s New Freedom Commission: recommendations
to transform mental health care in America. Psychiatr Serv.
2003;54:1467-1474.
8. Appendix S4: the case for routine mental health screening.
Pediatrics. 2010;125(suppl 3):S133-S139.
9. Wolraich ML, Lambert W, Doffing MA, Bickman L, Simmons
T, Worley K. Psychometric properties of the Vanderbilt
ADHD Diagnostic Parent Rating Scale in a referred popula-
tion. J Pediatr Psychol. 2003;28:559-568.
10. Johnson JG, Harris ES, Spitzer RL, Williams JB. The patient
health questionnaire for adolescents: validation of an instru-
ment for the assessment of mental disorders among adolescent
primary care patients. J Adolesc Health. 2002;30:196-204.
11. Jellinek M, Evans N, Knight RB. Use of a behavior checklist
on a pediatric inpatient unit. J Pediatr. 1979;94:156-158.
12. Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A
randomized controlled trial of screening for maternal depres-
sion with a clinical decision support system. J Am Med Inform
Assoc. 2013;20:311-316.
13. Murphy JM, Ichinose C, Hicks RC, et al. Utility of the
Pediatric Symptom Checklist as a psychosocial screen to
meet the federal Early and Periodic Screening, Diagnosis,
and Treatment (EPSDT) standards: a pilot study. J Pediatr.
1996;129:864-869.
14. Lavigne JV, Binns HJ, Christoffel KK, et al. Behavioral and
emotional problems among preschool children in pediatric pri-
mary care: prevalence and pediatricians’ recognition. Pediatric
Practice Research Group. Pediatrics. 1993;91:649-655.
15. Rushton J, Bruckman D, Kelleher K. Primary care referral of
children with psychosocial problems. Arch Pediatr Adolesc
Med. 2002;156:592-598.
16. Gilbody S, Sheldon T, House A. Screening and case-find-
ing instruments for depression: a meta-analysis. CMAJ.
2008;178:997-1003.
17. Wissow LS, Brown J, Fothergill KE, et al. Universal men-
tal health screening in pediatric primary care: a systematic
review. J Am Acad Child Adolesc Psychiatry. 2013;52:1134.
e23-1147.e23.
18. Jellinek MS, Murphy JM, Burns BJ. Brief psychoso-
cial screening in outpatient pediatric practice. J Pediatr.
1986;109:371-378.
19. Simonian SJ, Tarnowski KJ. Utility of the Pediatric Symptom
Checklist for behavioral screening of disadvantaged children.
Child Psychiatry Hum Dev. 2001;31:269-278.
20. Walker WO Jr, LaGrone RG, Atkinson AW. Psychosocial
screening in pediatric practice: identifying high-risk children.
J Dev Behav Pediatr. 1989;10:134-138.
21. Gardner W, Kelleher KJ, Pajer KA, Campo JV. Primary care
clinicians’ use of standardized tools to assess child psychoso-
cial problems. Ambul Pediatr. 2003;3:191-195.
22. Pidano AE, Kimmelblatt CA, Neace WP. Behavioral health in
the pediatric primary care setting: needs, barriers, and impli-
cations for psychologists. Psychol Serv. 2011;8:151-165.
23. Chung PJ, Lee TC, Morrison JL, Schuster MA. Preventive
care for children in the United States: quality and barriers.
Annu Rev Public Health. 2006;27:491-515.
24. Stevens GD, Seid M, Mistry R, Halfon N. Disparities in pri-
mary care for vulnerable children: the influence of multiple
risk factors. Health Serv Res. 2006;41:507-531.
25. Hacker K, Goldstein J, Link D, et al. Pediatric provider pro-
cesses for behavioral health screening, decision making, and
referral in sites with colocated mental health services. J Dev
Behav Pediatr. 2013;34:680-687.
26. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson
C, Mittman B. Implementation research in mental health
services: an emerging science with conceptual, method-
ological, and training challenges. Adm Policy Ment Health.
2009;36:24-34.
27. Van Cleave J, Kuhlthau KA, Bloom S, et al. Interventions to
improve screening and follow-up in primary care: a system-
atic review of the evidence. Acad Pediatr. 2012;12:269-282.
28. Weitzman CC, Leventhal JM. Screening for behavioral health
problems in primary care. Curr Opin Pediatr. 2006;18:641-648.
Author Biographies
Rachel J. Valleley, PhD, is an associate professor of Psychology
and Pediatrics at the Munroe-Meyer Institute and the University of
Nebraska Medical Center. Her main research interests involve
integrating behavioral health services into pediatric practices.
Natalie Romer, PhD, is an assistant professor in the Department
of Child and Family Studies at the University of South Florida.
Her main research interests include universal screening, strength-
based assessment, and implementation of multi-tiered behavior
and mental health support systems.
Sara Kupzyk, PhD, is an assistant professor of Psychology with
the Munroe-Meyer Institute at the University of Nebraska Medical
Center. She provides outpatient behavioral health services in pedi-
atric primary care clinics.
Joseph H. Evans is a professor and director of Psychology at the
Munroe-Meyer Institute and at the University of Nebraska Medical
Center. With training funds from the Health Resources and
Services Administration, he has developed an “integrated behav-
ioral health in pediatric primary care model” that is now being
used in 32 rural and urban sites across Nebraska and that is being
replicated in two additional states.  Dr. Evans specializes in train-
ing psychology interns and post-doctoral fellows in methods to
integrate behavioral services into patients’ primary care “medical
homes.”
Keith D. Allen, PhD, is a professor of Psychology and Pediatrics
at the Munroe-Meyer Institute and the University of Nebraska
Medical Center.  His interests include pediatric pain management,
health-related behavior in children, parent training, stress-related
disorders, and management of distress during invasive medical
and dental procedures.