I
PROGRAM
EVALUATION
INTEREST AND
SKILLS OF SCHOOL
COUNSELORS
School counselors participated in a study
examining their program evaluation
interest and skills. Findings suggest
that school counselors understand the
importance of program evaluation, yet
they may lack the skills and confidence to
successfully engage in program evaluation
activities. Professional development
training may be an important method
for helping today’s school counselors
develop skills and confidence in program
evaluation. This article presents
implications for school counselor training
and practice and recommendations for
future research.
54 ASCA | PROFESSIONAL SCHOOL COUNSELING
n the past 20 years, school counsel-
ors made considerable gains in their
professional identity. Developments
including the National Standards for
School Counseling Programs (Ameri-
can School Counselor Association
[ASCA], 1997) and the ASCA Nation-
al Model (ASCA, 2012) helped pro-
mote the importance of school coun-
seling in 21st-century schools. Integral
to these frameworks, accountability
and the use of data to demonstrate
effective outcomes became prominent
features of comprehensive school
counseling programs (Bemak, Wil-
liams, & Chung, 2014; Carey & Dim-
mitt, 2008; Dimmitt, 2009; Poynton
& Carey, 2006; Sink, 2009). Coincid-
ing with U.S. education reforms during
the 2000s, including the No Child Left
Behind Act (U.S. Department of Edu-
cation [USDOE], 2001), researchers
urged school counselors to document
the impact and effectiveness of their
school counseling services in order to
provide accountability to stakeholder
groups (e.g., Dahir & Stone, 2003;
Studer, Oberman, & Womack, 2006;
Young & Kaffenberger, 2011). There-
fore, today’s school counselors need
strong skills in data analysis as well as
an organized, accountability-focused
evaluation framework for examining
the effectiveness of their programs and
services (Astramovich, Hoskins, &
Coker, 2013; Maras, Coleman, Gys-
bers, Herman, & Stanley, 2013).
Randall L. Astramovich, Ph.D., is an
assistant professor in the Department
of Counseling at Idaho State University.
E-mail: astrrand@isu.edu
doi: 10.5330/1096-2409-20.1.54
CALLS FOR
ACCOUNTABILITY
AND USE OF DATA IN
SCHOOL COUNSELING
Researchers discussed accountabil-
ity in school counseling for several
decades (e.g., Borders & Drury, 1992;
Fairchild, 1993, Fairchild & Seeley,
1995; Fairchild & Zins, 1986; House
& Hayes, 2002; Hughes & James,
2001). Likewise, school counsel-
ing delivery models (ASCA, 2012;
Dollarhide & Saginak, 2012) and
school counseling accreditation stan-
dards (Council for Accreditation of
Counseling and Related Educational
Programs [CACREP], 2015) refer-
enced the need for school counselor
accountability. In school counseling,
accountability refers to the pro-
cess of providing stakeholders with
data-based information about the
outcomes and effectiveness of school
counseling programs and services,
with a particular emphasis on how
students are specifically impacted
by the work of the school counselor
(ASCA, 2012). Stakeholders of school
counseling programs typically include
students, parents, teachers, adminis-
trators, and the local community—
all potential constituents of school
counseling programs who may have a
high degree of interest in the success
of programs and services provided by
school counselors. Conceptualized as
a bridge between school counseling
programs and stakeholder groups, ac-
countability helps school counselors
market and promote the many ways
they enhance student achievement
and personal/social development (As-
tramovich & Coker, 2007; Astramov-
ich et al., 2013).
Accountability efforts may serve
many purposes; however, a primary
goal of school counselor account-
ability involves establishing school
counselor credibility in the education
environment (Young & Kaffenberger,
2011), especially during times of fis-
cal crises or budget cuts when school
personnel may be furloughed (Bemak
et al., 2014: Fairchild & Zins, 1986).
In addition to protecting school
counselor jobs, Sink (2009) suggested
that accountability practices may also
help to solidify the leadership roles
of school counselors, and House and
Hayes (2002) emphasized the role
of accountability in helping school
counselors to advocate for all students
and facilitate student success. At the
national level, both the No Child Left
Behind Act (USDOE, 2001) and the
Blueprint for Reform (USDOE, 2010)
sustained the emphasis on account-
ability in public education (DeLuca
& Bellara, 2013), thus ensuring that
accountability practices will remain a
critical component of comprehensive
school counseling programs in the
years to come.
So that school counselors could
effectively meet accountability ex-
pectations, researchers urged them to
develop strong data and evaluation
skills (e.g., Astramovich, Hoskins, &
Coker, 2005; Bauman, 2006; Brig-
man, 2006; Rowell, 2005, 2006;
Whiston, 1996), with a particular
emphasis on competence in analyzing
and utilizing program-related data.
ACCOUNTABILITY HELPS SCHOOL COUNSELORS MARKET
AND PROMOTE THE MANY WAYS THEY ENHANCE STUDENT
ACHIEVEMENT AND PERSONAL/SOCIAL DEVELOPMENT.
The ASCA National Model (ASCA,
2012) encouraged school counselors
to develop data-driven practices and
the use of data specifically within
the management and accountability
components of comprehensive school
counseling programs. Bemak et al.
(2014) suggested that, as part of their
accountability efforts, school counsel-
ors might find it beneficial to analyze
readily available school data includ-
ing grades, attendance, discipline, and
school suspensions. School counselors
might also specifically evaluate data
related to students’ life skills in the
academic, career, and social/emotional
domains (Sink, 2009). According to
ASCA (2012), school counselors draw
on a combination of process, percep-
tion, and outcome data in order to
provide program accountability and to
demonstrate the impact of the school
counselor on student success. Given
the heightened emphasis on data-
driven school counseling, program
evaluation may offer school counselors
a framework for organizing their use
of data. Therefore, the current study
aimed to specifically explore the pro-
gram evaluation interest and skills of
school counselors.
PROGRAM
EVALUATION IN
SCHOOL COUNSELING
Although much has been published
about the need for school counsel-
ors to use data for accountability
(e.g., ASCA, 2012; Dimmitt, 2009;
Fairchild, 1993, Fairchild & Seeley,
1995; Myrick, 2003; Vacc & Rhyne-
Winkler, 1993), without an organized
evaluation system, they may find it
difficult to conceptualize how to col-
lect, analyze, and interpret data in a
meaningful manner (Astramovich et
al., 2013; Carey & Dimmitt, 2008;
Sink, 2009). Counseling program
evaluation may therefore provide
school counselors with a framework
to help organize their collection and
application of program-related data
(Astramovich, Hoskins, & Erford,
2015). The ASCA National Model
(ASCA, 2012) specifically included
program evaluation as an integral
part of the accountability component
of comprehensive school counseling
programs, thus underscoring the need
for today’s school counselors to have
VOLUME 20, NUMBER 1 | ASCA 55

competence in program evaluation
techniques.
Counseling program evaluation
involves the organized use of evalua-
tion techniques to monitor, assess, and
analyze the effectiveness and outcomes
of counseling services (Astramovich
& Coker, 2007). Specifically in the
school setting, the process of coun-
seling program evaluation generally
includes needs assessments, program
development and delivery, outcomes
assessment, and accountability and
advocacy with stakeholders (Astra-
movich et al., 2013). Counselor-driven
program evaluations typically involve
smaller sample sizes than large-scale
educational evaluations that may be
conducted by a team of researchers.
Nielson (2015) suggested that coun-
selors could benefit from purposefully
utilizing program evaluation tech-
niques with smaller, single-case designs
as a means for providing evidence-
based services to clients.
EVALUATION MAY OFFER SCHOOL COUNSELORS A
FRAMEWORK FOR ORGANIZING THEIR USE OF DATA.
School counselors historically
avoided program evaluation ac-
tivities, often as a result of a lack of
knowledge, skills, and confidence in
conducting evaluations (Maras et al.,
2013). School counselors also may
hesitate to evaluate their services out
of apprehension that the findings
may suggest that various programs
and services fail to provide satisfac-
tory results. Despite these concerns,
Astramovich et al. (2005) noted that
school counselors generally report
high levels of interest in evaluating
their programs and services, but they
may lack training in program evalua-
tion methods. Thus, school counsel-
ors may need increased opportunities
for training in data analysis and
program evaluation. Ultimately, an
organized and systematic program
evaluation process can provide school
counselors with critical data to help
ensure the best services are being
provided to students.
56 ASCA | PROFESSIONAL SCHOOL COUNSELING
STUDIES EXAMINING
DATA AND PROGRAM
EVALUATION IN
SCHOOL COUNSELING
Despite the frequent calls for data-
driven and accountable school
counseling programs (e.g., ASCA,
2012; Astramovich et al., 2013; Dahir
& Stone, 2003; Young & Kaffen-
berger, 2011), the literature includes
only a few studies examining school
counselors’ interest and skills in using
data and program evaluation. Early
researchers often explored school
counselors’ use of data related to stan-
dardized tests and assessments (e.g.,
Ekstrom, Elmore, Schafer, Trotter, &
Webster, 2004; Elmore & Ekstrom,
1993). More recently, researchers ap-
pear focused on examining data and
program evaluation skills necessary to
organize data-driven school counsel-
ing programs as outlined in the ASCA
National Model (ASCA, 2012; e.g.,
Maras et al., 2013; Poynton, 2009;
Young & Kaffenberger, 2011).
Research findings vary among recent
studies about school counselors’ use
of data, possibly due to differences
among the samples involved. Hol-
comb-McCoy, Gonzalez, and Johnston
(2009) found a low level of reported
data use among a general population
of school counselors even though
participants were from districts that
provided professional development on
the use of data. In contrast, Young and
Kaffenberger (2011) studied school
counselors in Recognized ASCA Mod-
el Programs (RAMP) and discovered
very high levels of data usage related
to program decision making. These
results are not surprising given that
RAMP school counselors are expected
to document their use of program
data in the RAMP application process
(ASCA, 2012).
Results also differ among studies of
professional development for enhanc-
ing school counselors’ use of data.
Young and Kaffenberger (2015) found
that professional development training
specifically on the use of program data
was an effective means for helping
counselors build data analysis skills.
However, Holcomb-McCoy et al.
(2009) suggested that professional
development activities about pro-
gram data were not associated with
increased utilization of data by school
counselors. Similarly, Poynton (2009)
noted that school counselors partici-
pating in professional development
trainings on the use of technology for
managing data did not sustain their
use of data after the trainings, primar-
ily due to reported time constraints.
Other researchers specifically ex-
plored program evaluation as a frame-
work for school counselors’ use of
data and accountability efforts. Studies
by Astramovich et al. (2005) and
Maras et al. (2013) found high levels
of interest and value in program evalu-
ation among school counselors. How-
ever, confidence in evaluation skills
appeared to vary, with higher levels
of evaluation competence reported
by those school counselors involved
in a statewide mentoring program
(Maras et al., 2013). Furthermore,
lack of previous training in evaluation
methods (Astramovich et al., 2005)
and statistical skill levels (Maras et
al., 2013) may affect the proclivity of
school counselors to involve them-
selves in data-based program evalua-
tion activities.
Overall, previous research suggests
that school counselors generally rec-
ognize the value in utilizing data and
evaluating their programs and services,
but their data skills and usage varies
widely. Of the limited research to date,
some studies involved RAMP counsel-
ors or school counselors in mentoring
programs and thus current literature
findings may not accurately reflect the
interest and skills in data and evalua-
tion among more general populations
of school counselors. Furthermore,
few studies specifically addressed the
degree to which school counselors
desire training in program evaluation
skills and uncertainties persist about
the influence of professional develop-
ment trainings on enhancing school
counselors’ use of data and program
evaluation.
RESEARCH
QUESTIONS
Accountability, use of data, and
program evaluation remain consis-
tent themes in the school counseling
literature. However, inconsistent find-
ings among the few previous studies
suggest the need for further explora-
tion of school counselors’ program
evaluation skills. Additional research
may also help identify the importance
that school counselors place on receiv-
ing training in program evaluation and
help clarify the influence of profession-
al development on facilitating program
evaluation competence. Given these
needs, a study was conducted with the
following guiding research questions:
1. How interested are elementary,
middle, and high school counsel-
ors in program evaluation?
2. How do elementary, middle, and
high school counselors rate their
current program evaluation skills
and confidence in conducting
program evaluations?
3. What importance do elementary,
middle, and high school counsel-
ors place on program evaluation
training?
4. How does prior professional
development training in program
evaluation methods affect program
evaluation interest and self-reported
skills among elementary, middle,
and high school counselors?
METHOD
Participants
Power analysis helps researchers
minimize the chance of making a Type
II error (Balkin & Sheperis, 2011).
For this study, the author conducted
a power analysis utilizing G*Power
statistics software (Faul, Erdfelder,
Lang, & Buchner, 2007) assuming an
alpha level of .05, power at .95, and a
medium effect size of .15. The power
analysis suggested that a minimum
sample size of 135 was needed for
using Multiple Analysis of Variance
(MANOVA) procedures. Therefore,
the sample of 241 school counselors in
the study was determined to provide
appropriate statistical power.
School counselors (N = 241) from
a large school district in the West-
ern U.S. participated in this study.
The district includes more than 300
schools and serves approximately
300,000 students in urban, subur-
ban, and rural settings, with 61% of
district students being from ethnic
minority populations. Two schools
in the district are designated as
Recognized ASCA Model Programs
(RAMPs). The sample included 185
(76.8%) females and 54 (22.4%)
males (2 participants [1%] did not
specify gender) with a mean age of
43.4 years (SD = 10.1) and an aver-
age of 7.7 years (SD = 6.6) of school
AN ORGANIZED AND SYSTEMATIC PROGRAM
EVALUATION PROCESS CAN … HELP ENSURE THE BEST
SERVICES ARE BEING PROVIDED TO STUDENTS.
counseling experience. Ethnicity of
the participants included 191 (79.3%)
Caucasians, 21 (8.7%) African
Americans, 14 (5.8%) Latinos and
Hispanics, 3 (1.2%) Asian Americans,
2 (0.8%) Native Americans, and 6
(2.4%) indicating other or mixed
ethnic background; 4 participants
(1.7%) did not specify ethnicity.
School levels included 69 (28.6%)
elementary, 72 (29.9%) middle, and
96 (39.8%) high school counselors;
4 participants (1.7%) did not specify
school level. The participants’ school
setting included 143 (59.3%) ur-
ban, 78 (32.4%) suburban, and 7
(2.9%) rural school counselors; 13
participants (5.4%) did not specify
setting. Ninety-six (39.8%) of the
participants indicated having received
professional development workshop
training in counseling program evalu-
ation methods during the previous 12
months, either at professional confer-
ences or offered through the school
district.
Instrumentation
Only one research instrument exam-
ining school counselors’ use of data
and program evaluation was pub-
lished prior to this study (Maras et
al., 2013). However, items from the
Maras et al. Effective Practices Survey
did not consistently utilize terminol-
ogy as identified in published models
of counseling program evaluation
(i.e., Astramovich, Hoskins, & Erford,
2015) and it included items on the use
of specific statistical tests (e.g., t tests
and correlations), which were not part
of the current study. Furthermore, the
instrument did not assess for school
counselors’ interest in program evalua-
tion training. Consequently, the Maras
et al. Effective Practices Survey (2013)
was deemed inappropriate for use in
the current study.
Because no published measures
aligned with the goals of this study,
the author developed a 20-item instru-
ment, the Program Evaluation Inter-
est and Skills Assessment (PEISA), to
explore the primary research ques-
tions. Based on counseling program
evaluation skills identified by ASCA
(2012) and Astramovich et al. (2015),
the PEISA contains 20 items assessing
school counselors’ program evalua-
tion interest and skills using a 5-point
Likert-type scale (1 = strongly disagree
to 5 = strongly agree). Prior to its
use, two school counselor educators
and two practicing school counselors
evaluated the content validity of the
PEISA and determined that the items
VOLUME 20, NUMBER 1 | ASCA 57
 TABLE 1 PRINCIPAL COMPONENTS ANALYSIS WITH VARIMAX
ROTATION COMPONENT LOADINGS FOR THE PEISA
Component
Item 1 2 3 4
I know how to utilize stakeholder feedback to inform my school counseling program .85 .17 .05 .23
I know how to utilize outcome data to modify my school counseling program .84 .13 .10 .15
I know how to analyze data to evaluate program outcomes .83 -.12 .15 .19
I know how to collect data for program evaluations .83 -.06 .12 .18
I know how to communicate results of program evaluations to stakeholder groups .79 .04 .09 .24
I know how to conduct needs assessments .77 .10 .04 .03
I know how to identify program components to be evaluated .73 .08 .04 .22
I know how to analyze data utilizing computer software .72 -.23 .08 .14
I am interested in conducting an evaluation of my school counseling program .05 .86 .21 .14
I would like to formally evaluate the success of my school counseling services -.04 .86 .24 .13
I would like to receive training in school counseling program evaluation methods -.09 .79 .31 -.15
I am interested in developing my skills in program evaluation -.06 .79 .33 -.15
Conducting an evaluation of my school counseling program is important .17 .75 .31 .09
School counselor education programs should train students to conduct .13 .32 .84 .12
program evaluations
Hands-on evaluation experience should be included in school counselor .07 .39 .81 .10
education programs
Future school counselors should understand how to evaluate counseling programs .07 .40 .79 -.02
Future school counselors should understand how to utilize program evaluation data .23 .33 .78 .05
I could explain to another school counselor how to evaluate the success .49 .00 .04 .75
of counseling programs
I understand the process of conducting an evaluation of my school .27 -.02 .24 .76
counseling program
I am confident in my current ability in program evaluation .48 .10 -.10 .68

Note. Factor loadings >.60 are in boldface.

seemed appropriate for the aims of the
study. Items from the PEISA are found
in Table 1.
The author conducted a principal
components analysis (PCA) with vari-
max rotation and Kaiser normaliza-
tion to detect orthogonal components
in the PEISA. The PCA method was
selected with the intent of creating
composite scores for any identified
components, with no expectation
regarding the potential number of
detected components. According to
Tabachnick and Fidell (2012), the
sample of 241 exceeded the minimum
suggested sample size for a 20-item
instrument and a PCA solution with
high factor loadings (> .80) gener-
ally reduces the minimum acceptable
sample size to about 150 (Mertler &
Vannatta, 2002). Furthermore, the
Kaiser-Meyer-Olkin test of sampling
adequacy (KMO = .89) and Bartlett’s
test of sphericity (χ2 (190) = 3245.97,
p < .05) suggested that the PEISA
items contained high levels of com-
monalities appropriate for conduct-
ing a PCA. Four components with
eigenvalues greater than 1.0 were
extracted (see Table 1). Component
one (eigenvalue = 7.32) accounted for
36.64% of the variance. Component
two (eigenvalue = 5.21) accounted for
26.07% of the variance. Component
three (eigenvalue = 1.18) accounted
for 5.88% of the variance. Component
four (eigenvalue = 1.06) accounted for
5.28% of the variance. The four fac-
tors combined accounted for 73.87%
of the variance.
Examining the items that loaded
in each component helped identify
descriptive labels. Component one
contained eight items that appeared to
measure program evaluation skill level
(Cronbach’s α = .93). Component two
contained eight items that appeared to
measure interest in program evalua-
tion (Cronbach’s α = .91). Component
three contained four items appearing
to measure training importance of pro-

58 ASCA | PROFESSIONAL SCHOOL COUNSELING

gram evaluation in school counseling
(Cronbach’s α = .92). Component four
contained three items and appeared to
measure confidence in conducting pro-
gram evaluation (Cronbach’s α = .81).
Component themes identified in the
PEISA were therefore determined ap-
propriate for the research questions
posed in the study.
In addition to the PEISA, partici-
pants in the study completed a demo-
graphic questionnaire that gathered in-
formation including age, sex, ethnicity,
years experience as a school counselor,
grade level (elementary, middle, or
high school), location of current em-
ployment (urban, suburban, or rural
setting), and whether the participant
had received prior training in counsel-
ing program evaluation methods in
professional development workshops
during the previous 12 months, either
at professional conferences or offered
by the school district.
Procedures
Participants were invited to take part
in the study at the start of a required
professional development training day
for district school counselors. After
the purpose of the study and Institu-
tional Review Board approval were re-
viewed, participants received a verbal
definition of program evaluation based
on the ASCA National Model (ASCA,
2012) and were asked to complete the
instrument packets. All instrument
packets were completed and collected
prior to the opening professional
development session focused on school
counselor accountability. A total of
290 instrument packets were distrib-
uted. Of these, participants returned
241 useable packets, resulting in an
83% participation rate.
RESULTS
To explore the first three research
questions regarding program evalu-
ation interest, skill levels and confi-
dence, and training importance, the
author calculated composite scores
for each of the four components
of the PEISA by averaging the re-
sponses to all items in the component.
For program evaluation skill level,
participants scored a mean of 3.01
(SD = .93). For interest in program
evaluation, participants scored a
mean of 3.83 (SD = .83). For training
importance of program evaluation,
participants scored a mean of 4.19
(SD = .69). For confidence in conduct-
ing program evaluation, participants
scored a mean of 3.05 (SD = .87).
A DATA MENTOR COULD HELP SCHOOL COUNSELORS
DEVELOP SKILLS AND CONFIDENCE FROM
SOURCES OTHER THAN ATTENDING PROFESSIONAL
DEVELOPMENT WORKSHOPS.
Pairwise comparisons of the PEISA
component scores (with Bonferroni cor-
rections) found several significant dif-
ferences. Program evaluation skill level
was significantly lower than interest
in program evaluation (t(204) = -9.10,
p < .000 (2-tailed), η2 = .32) and
program evaluation skill level was
significantly lower than training
importance (t(212) = -17.07, p < .000
(2-tailed), η2 = .58). Interest in program
evaluation was significantly lower than
training importance (t(219) = -8.44,
p < .000 (2-tailed), η2 = .25) and
interest in program evaluation was
significantly higher than confidence
in conducting program evaluation
(t(219) = 10.05, p < .000 (2-tailed),
η2 = .32). Last, training importance
was significantly higher than confidence
in conducting program evaluation
(t(224) = 16.71, p < .000 (2-tailed),
η2 = .55). These results indicate that
participants appear to have high levels
of interest in program evaluation and
place high value on training in program
evaluation. However, their current con-
fidence and skills in program evaluation
are rated lower.
To explore the fourth research
question regarding professional
development impact on interest and
skills in program evaluation, the
author conducted a 2 (prior profes-
sional development program evalua-
tion training: yes or no) x 3 (school
level: elementary, middle, or high)
factorial multiple analysis of variance
(MANOVA) with the four dependent
variables (i.e., program evaluation skill
level, interest in program evaluation,
training importance, and confidence
in conducting program evaluations).
The data was screened for outliers
and participants who were missing
responses for any items were removed
from this analysis (n = 49). A test of
normality (Kolmogorov-Smirnov) sug-
gested that three of the four dependent
variables were non-normally distrib-
uted; however, MANOVA procedures
are considered robust to non-normal-
ity (Mertler & Vannatta, 2002) and
therefore the data was not subjected to
transformations. Box’s Test for homo-
geneity of covariance was significant
(Box’s M = 102.60, p = .000); there-
fore, Pillai’s Trace statistic was used
for the multivariate tests.
For the fourth research question,
the multivariate test for prior pro-
gram evaluation professional devel-
opment training main effect found
significant differences, Pillai’s V = .12,
F(4, 183) = 6.43, p < .001, ηp2 = .12.
Univariate analysis of variance
(ANOVA) follow-up tests indicated
that participants who had received
prior professional development
training in program evaluation had
significantly higher scores than those
who had no previous professional
development training on the four
dependent variables: program evalu-
ation skill level, F(1,186) = 14.82,
p < .001, ηp2 = .07; interest in program
evaluation, F(1,186) = 4.61, p < .05,
ηp2 = .02; training importance in
program evaluation, F(1,186) = 6.26,
VOLUME 20, NUMBER 1 | ASCA 59
 TABLE 2 MEANS AND STANDARD DEVIATIONS FOR DEPENDENT VARIABLES BY FACTORS
Factor
Prior
School Professional
Dependent Variable Level Development n M SD
Program Evaluation Skill Level Elementary No 28 2.54 .91
Yes 29 3.25 1.04
Middle No 31 2.74 1.05
Yes 25 3.25 .89
High No 50 3.00 .92
Yes 29 3.35 .66
Total No 109 2.80 .97
Yes 83 3.28 .87
Interest in Program Evaluation Elementary No 28 3.77 .76
Yes 29 3.98 1.02
Middle No 31 3.81 .75
Yes 25 4.14 .62
High No 50 3.62 .99
Yes 29 3.88 .67
Total No 109 3.71 .87
Yes 83 3.99 .80
Program Evaluation Training Importance Elementary No 28 4.11 .89
Yes 29 4.53 .52
Middle No 31 3.97 .73
Yes 25 4.36 .66
High No 50 4.08 .67
Yes 29 4.37 .60
Total No 109 4.06 .74
Yes 83 4.42 .59
Confidence in Conducting Program Evaluation Elementary No 28 2.74 .73
Yes 29 3.43 1.08
Middle No 31 2.83 .72
Yes 25 3.28 .82
High No 50 2.97 .83
Yes 29 3.22 .78
Total No 109 2.87 .78
Yes 83 3.31 .90

p < .001, ηp2 = .07; and confidence
in conducting program evaluation,
F(1,186) = 14.10, p < .001, ηp2 = .07.
The multivariate test for school level
(i.e., elementary, middle, high) was
not statistically significant, Pillai’s
V = .07, F(8, 368) = 1.81, p = .12. The
multivariate test for interaction effects
between prior professional develop-
ment training in program evaluation
and school level also was not statisti-
cally significant, Pillai’s V = .02, F(8,
368) = .34, p = .95. Therefore, interest
in program evaluation, program
evaluation skill level, confidence in
conducting program evaluation, and
importance of training were all rated
significantly higher by participants

60 ASCA | PROFESSIONAL SCHOOL COUNSELING

who had participated in professional
development in the past 12 months.
Table 2 provides the means and
standard deviations of the dependent
variables by each factor.
DISCUSSION
The calls for school counselors to
develop data-driven school counsel-
ing programs increased in recent years
(e.g., ASCA, 2012). Today’s school
counselors, therefore, need strong
program evaluation and data analysis
skills to effectively meet accountability
demands, identify best practices in
school counseling, and advocate both
for students and for the profession of
school counseling. This study sought
to examine program evaluation inter-
est, skill levels, and training impor-
tance among school counselors; it also
aimed to explore the influence of prior
professional development training in
program evaluation on school coun-
selors’ interest and skills in program
evaluation.
Program Evaluation Interest,
Skills, and Training Importance
Participants in this study rated their
interest in program evaluation and the
importance of school counselor train-
ing in program evaluation methods
significantly higher than their current
program evaluation skill levels and
confidence in conducting program
evaluation. The finding of a high level
of interest in program evaluation
is consistent with previous studies
on school counselors’ use of data
and evaluation (e.g., Maras et al.,
2013; Young & Kaffenberger, 2015)
and suggests that school counselors
generally see the value in program
evaluation. The author also found
that school counselors at all school
levels place high levels of importance
on training in program evaluation.
Previous research on program evalu-
ation training importance was limited
and only utilized a small sample of
elementary school counselors (Astra-
movich et al., 2005). Therefore, this
study helped demonstrate that school
counselors at all school levels appear
to view training in program evalua-
tion as important. The use of data for
accountability and the use of pro-
gram evaluations are emphasized for
counselors at all school levels in the
ASCA National Model (ASCA, 2012),
although interventions and programs
being evaluated may differ based on
the particular school level and set-
ting. Based on these results, school
counselors may benefit from program
evaluation training that helps them to
identify appropriate program elements
to be evaluated for their particular
school level and setting.
TODAY’S SCHOOL COUNSELORS ARE EXPECTED TO HAVE
MORE SOPHISTICATED EVALUATION AND DATA ANALYSIS
SKILLS THAN IN PREVIOUS DECADES.
Regarding program evaluation
skills, the present study found lower
self-reported program evaluation skill
levels among participants. This finding
differs from prior research on school
counselors’ data skill levels (Young
& Kaffenberger, 2011) and program
evaluation skill levels (Maras et al.,
2013), perhaps due to differences in
the samples of school counselors being
studied. The current study utilized a
general sample of school counselors
from a large school district, whereas
Young and Kaffenberger (2011)
studied RAMP school counselors and
Maras et al. (2013) examined school
counselors involved in a mentoring
program. School counselor popula-
tions involved in mentoring or work-
ing in RAMP schools are likely to uti-
lize data and program evaluation on a
regular basis and have higher program
evaluation skills and confidence. How-
ever, findings from this study indicated
that participants from a general popu-
lation of school counselors reported
lower levels of program evaluation
skills and confidence. As pointed out
by Holcomb-McCoy et al. (2009),
today’s school counselors are expected
to have more sophisticated evaluation
and data analysis skills than in previ-
ous decades. Consequently, school
counselors, especially those not in
RAMP schools or involved in mentor-
ing programs, may need frequent pro-
fessional development opportunities to
learn and reinforce data analysis and
program evaluation skills.
Professional Development
Training in Program Evaluation
A key discovery in this study involved
the differences found when comparing
PEISA scores among the participants
who had received professional devel-
opment training in program evaluation
during the previous 12 months with
those who had no such training. The
medium effect size (Vacha-Haase &
Thompson, 2004) suggests that the
school counselors who had par-
ticipated in professional development
training in program evaluation in the
prior 12 months reported somewhat
higher interest and skill levels in pro-
gram evaluation and more confidence
in their abilities to engage in evalua-
tion activities. However, fewer than
half of the school counselors in the
sample indicated they had participated
in professional development train-
ing in program evaluation during the
previous 12 months. Similar to results
reported by Young and Kaffenberger
(2015), these findings underscore the
potential importance of professional
development training in helping school
counselors develop competence in
utilizing data and program evaluation
as part of their professional practice.
However, findings from the current
study contrast with other studies on
the effectiveness of professional de-
velopment on the use of data. Hol-
comb-McCoy et al. (2009) found the
frequency of professional development
activities to be unassociated with the
VOLUME 20, NUMBER 1 | ASCA 61
REGULAR PROFESSIONAL DEVELOPMENT OPPORTUNITIES …
IN PROGRAM EVALUATION AND DATA ANALYSIS COULD
HELP CURRENT SCHOOL COUNSELORS DEVELOP DATA-
DRIVEN SCHOOL COUNSELING SERVICES.
use of program data by school coun-
selors and Poynton (2009) also found
professional development trainings did
not improve school counselors’ use of
data. As suggested by Holcomb-Mc-
Coy et al. (2009), improving the qual-
ity of school counselors’ professional
development trainings, specifically
with the inclusion of peer coaching
and mentoring, may be essential to
help school counselors gain confidence
in the analysis and application of
program data.
Limitations & Implications for
Professional School Counseling
Findings of this study should be
viewed with regard to its limitations.
The participants comprised a conve-
nience sample of school counselors
from one large school district in the
Western U.S.; thus, the results may not
represent other populations of school
counselors. Furthermore, the study
relied on the self-report of evaluation
skills by the school counselor partici-
pants, and this may not provide an
accurate measure of program evalu-
ation abilities. The content, delivery
methods, and duration of participants’
previous program evaluation-specific
professional development train-
ings were not examined as part of
this study. Last, the PEISA is a new
measure of school counseling pro-
gram evaluation interest and skills
and although the current study found
evidence of internal reliability, further
research is needed to help assure con-
struct validity for the instrument.
Despite the limitations, the findings
of this study suggest several implica-
tions for school counselors, school
district counselor supervisors, and
counselor education programs. First,
regular professional development op-
portunities that facilitate the learning
62 ASCA | PROFESSIONAL SCHOOL COUNSELING
of skills in program evaluation and
data analysis could help current school
counselors develop data-driven school
counseling services that are aligned
with the recommendations of the
ASCA National Model (ASCA, 2012)
and may ultimately help contribute
to the research base of professional
school counseling (Kaffenberger,
2012). Rather than presenting data
analysis as an isolated activity, school
counselors also may benefit from
training specifically about program
evaluation so that the identification
and analysis of data can be understood
in the context of the larger evaluation
process (Astramovich et al., 2013;
Carey & Dimmitt, 2008; Sink, 2009).
As a follow-up to professional de-
velopment trainings, school counselors
may also benefit from participating in
mentoring programs specifically de-
signed to help reinforce data analysis
and program evaluation skills at their
specific school site. The development
of data analysis and program evalu-
ation skills may be conceptualized as
an ongoing developmental process;
therefore, regular contact with a data
mentor could help school counselors
develop skills and confidence from
sources other than attending profes-
sional development workshops. The
school counseling profession included
mentoring, particularly with new
school counselors, for many years
(e.g., Desmond, West, & Bubenzer,
2006; VanZandt & Perry, 1992).
Organized data mentoring programs
could therefore serve as an effective
strategy for helping today’s profession-
al school counselors become proficient
in designing, implementing, and evalu-
ating data-driven school counseling
programs.
In counselor education programs,
future school counselors may benefit
from more rigorous program evalua-
tion and data analysis training as part
of their graduate programs of study.
Research methods and assessment
classes may be used to help future
school counselors learn the fundamen-
tals of data analysis and the process
of evaluating counseling programs.
Program evaluation and accountability
activities also could be integrated into
students’ fieldwork experiences to help
reinforce the application of these skills
in an actual school setting. Counselor
educators and school district counselor
supervisors could collaborate on the
design and delivery of professional
development trainings and mentoring
programs on program evaluation and
data analysis that would address the
unique needs and concerns of counsel-
ors in the specific school district.
Results of this study also suggest the
need for further research on how best
to train school counselors in program
evaluation and data analysis. Future
studies could specifically examine the
content, duration, and frequency of
professional development trainings
related to program evaluation and the
use of data in order to clarify the most
effective professional development
methods for helping school counsel-
ors develop these skills. Other studies
designed to identify and explore the
specific evaluation and data analysis
skills necessary for successful school
counseling programs could help pri-
oritize training components and lead
to the development of school coun-
seling program evaluation and data
analysis curricula for use in counselor
education programs and professional
development workshops. Last, stud-
ies that examine data avoidance and
data anxiety among school counselors
may help identify effective interven-
tions and training methods that could
help school counselors gain more
confidence in their data analysis and
program evaluation abilities.
In today’s educational environments,
school counselors increasingly are
expected to demonstrate proficiency
in program evaluation in order to ef-
fectively plan and deliver data-driven
school counseling services. Helping
school counselors develop competence
and confidence in their program evalu-
ation skills, including the appropriate
analysis and use of program-related
data, appears central for the advance-
ment of the profession. To that end,
professional development training
opportunities show promise as an
effective means for helping school
counselors develop the skills in evalu-
ation and data analysis necessary for
successful 21st-century school counsel-
ing programs. n
REFERENCES
American School Counselor Association.
(1997). The national standards for
school counseling programs.
Alexandria, VA: Author.
American School Counselor Association.
(2012). The ASCA National Model: A
framework for school counseling
programs (3rd ed). Alexandria, VA:
Author.
Astramovich, R. L., & Coker, J. K. (2007).
Program evaluation: The Accountability
Bridge model for counselors. Journal of
Counseling & Development, 85,
162-172.
doi:10.1002/j.1556-6678.2007.tb00459.x
Astramovich, R. L., Coker, J. K., & Hoskins,
W. J. (2005). Training school counselors
in program evaluation. Professional
School Counseling, 9, 49-54.
doi:10.5330/prsc.9.1.80442r1q5n29x546
Astramovich, R. L., Hoskins, W. J., &
Coker, J. K. (2013). Organizing and
evaluating data-driven school
counseling programs (2nd ed.).
Dubuque, IA: Kendall Hunt.
Astramovich, R. L., Hoskins, W. J., &
Erford, B. T. (2015). Program evaluation
and accountability. In B. Erford (Ed.),
Research and evaluation in counseling
(2nd ed., pp. 198-216). Stanford, CT:
Cengage.
Balkin, R. S., & Sheperis, C. J. (2011).
Evaluating and reporting statistical
power in counseling research. Journal
of Counseling & Development, 89,
268-272.
doi:10.1002/j.1556-6678.2011.tb00088.x
Bauman, S. (2006). Using comparison
groups in school counseling research: A
primer. Professional School Counseling,
9, 357-366.
doi:10.5330/prsc.9.4.yn291788r41n0455
Bemak, F., Williams, J. M., & Chung, R. C.
(2014). Four critical domains of
accountability for school
counselors. Professional School
Counseling, 18, 100-110.
doi:10.5330/prsc.18.1.q40379257k35n1kx
Borders, L. D., & Drury, S. M. (1992).
Comprehensive school counseling
programs: A review for policymakers
and practitioners. Journal of Counseling
& Development, 70, 487-498.
doi:10.1002/j.1556-6676.1992.tb01643.x
Brigman, G. (2006). Research methods in
school counseling: A summary for the
practitioner. Professional School
Counseling, 9, 421-425.
doi:10.5330/prsc.9.4.ku4827rxx0558467
Carey, J., & Dimmitt, C. (2008). A model
for evidence-based elementary school
counseling: Using school data,
research, and evaluation to enhance
practice. The Elementary School
Journal, 108, 422-430.
doi:10.1086/589471
Council for Accreditation of Counseling
and Related Educational Programs.
(2015). 2016 CACREP standards.
Retrieved from http://www.cacrep.org/
wp-content/uploads/2012/10/2016-
Standards-with-Glossary.pdf
Dahir, C. A., & Stone, C. B. (2003).
Accountability: A M.E.A.S.U.R.E. of the
impact school counselors have on
student achievement. Professional
School Counseling, 6, 214-221.
Retrieved from http://www.jstor.org/
stable/42732431
Desmond, K. J., West, J. D., & Bubenzer,
D. L. (2006). Enriching the profession of
school counselling by mentoring novice
school counsellors without teaching
experience. Guidance & Counseling, 21,
174-183.
DeLuca, C., & Bellara, A. (2013). The
current state of assessment education:
Aligning policy, standards, and teacher
education curriculum. Journal of
Teacher Education, 64, 356-372.
doi:10.1177/0022487113488144
Dimmitt, C. (2009). Why evaluation
matters: Determining effective school
counseling practices. Professional
School Counseling, 12, 395-399.
doi:10.5330/PSC.n.2010-12.395
Dollarhide, C. T., & Saginak, K. A. (2012).
Comprehensive school counseling
programs: K–12 delivery systems in
action (2nd ed.). Upper Saddle River,
NJ: Pearson.
Elmore, P. B., & Ekstrom, R. B. (1993).
School counselors’ test use patterns
and practices. The School Counselor,
41, 73-80. Retrieved from http://www.
jstor.org/stable/23900097
Ekstrom, R. B., Elmore, P. B., Schafer, W.
D., Trotter, T. V., & Webster, B. (2004). A
survey of assessment and evaluation
activities of school
counselors. Professional School
Counseling, 8, 24-30. Retrieved from
http://www.jstor.org/stable/42732411
Fairchild, T. N. (1993). Accountability
practices of school counselors: 1990
national survey. The School Counselor,
40, 363-374. Retrieved from http://www.
jstor.org/stable/23900206
Fairchild, T. N., & Seeley, T. J. (1995).
Accountability strategies for school
counselors: A baker’s dozen. The
School Counselor, 42, 377-392.
Retrieved from http://www.jstor.org/
stable/23901012
Fairchild, T. N., & Zins, J. E. (1986).
Accountability practices of school
counselors: A national survey. Journal
of Counseling & Development, 65,
196-199.
doi:10.1002/j.1556-6676.1986.tb01313.x
Faul, F., Erdfelder, E., Lang, A.-G., &
Buchner, A. (2007). G*Power 3: A
flexible statistical power analysis
program for the social, behavioral, and
biomedical sciences. Behavior Research
Methods, 39, 175-191.
doi:10.3758/BF03193146
Holcomb-McCoy, C., Gonzalez, I., &
Johnston, G. (2009). School counselor
dispositions as predictors of data
usage. Professional School
Counseling, 12, 343-351.
doi:10.5330/PSC.n.2010-12.343
House, R. M., & Hayes, R. L. (2002).
School counselors: Becoming key
players in school reform. Professional
School Counseling, 5, 249-256.
Retrieved from http://www.
biomedsearch.com/article/School-
counselors-becoming-key-
players/86059885.html
Hughes, D. K., & James, S. H. (2001).
Using accountability data to protect a
school counseling program: One
counselor’s experience. Professional
School Counseling, 4, 306-309.
Retrieved from https://www.questia.
com/library/journal/1P3-70579298/
using-accountability-data-to-protect-a-
school-counceling
Kaffenberger, C. J. (2012). A call for school
counseling practitioner research.
Professional School Counseling, 16,
59-62. doi:10.5330/PSC.s.2010-12.392
Maras, M. A., Coleman, S. L., Gysbers, N.
C., Herman, K. C., & Stanley, B. (2013).
Measuring evaluation competency
among school counselors. Counseling
Outcome Research & Evaluation, 4,
99-111. doi:10.1177/2150137813494765
VOLUME 20, NUMBER 1 | ASCA 63
 Mertler, C. A., & Vannatta, R. A. (2002).
Advanced and multivariate statistical
methods: Practical application and
interpretation (2nd ed.). Los Angeles,
CA: Pyrczak.
Myrick, R. D. (2003). Accountability:
Counselors count. Professional School
Counseling, 6, 174-179. Retrieved from
http://www.jstor.org/stable/42732426
Nielson, T. (2015). Practice-based research:
Meeting the demands of program
evaluation through the single-case
design. Journal of Mental Health
Counseling, 37, 364-376.
doi:10.17744/mehc.37.4.07
Poynton, T. A., & Carey, J. C. (2006). An
integrative model of data-based
decision making for school counseling.
Professional School Counseling, 10,
121-130.
doi:10.5330/prsc.10.2.r5616876nt044766
Poynton, T. A. (2009). Evaluating the
effectiveness of a professional
development workshop to increase
school counselors’ use of data: The role
of technology. Journal of Counselor
Preparation & Supervision, 1, 30-49.
doi:10.7729/11.0107
Rowell, L. L. (2005). Collaborative action
research and school counselors.
Professional School Counseling, 9,
28-36.
doi:10.5330/prsc.9.1.x6302t28qp670x65
Rowell, L. L. (2006). Action research and
school counseling: Closing the gap
between research and practice.
Professional School Counseling, 9,
376-384.
doi:10.5330/prsc.9.4.g777740821404674
64 ASCA | PROFESSIONAL SCHOOL COUNSELING
Sink, C. A. (2009). School counselors as
accountability leaders: Another call for
action. Professional School Counseling,
13, 68-74. doi:10.5330/PSC.n.2010-13.68
Studer, J. R., Oberman, A. H., & Womack,
R. H. (2006). Producing evidence to
show counseling effectiveness in the
schools. Professional School
Counseling, 9, 385-391.
doi:prsc.9.4.106k25443020h5g7
Tabachnick, B. G., & Fidell, L. S. (2012).
Using multivariate statistics (6th ed.).
Boston, MA: Allyn & Bacon.
U. S. Department of Education. (2001). No
Child Left Behind Act of 2001 (H.R.1).
Washington, DC: Author.
U.S. Department of Education. (2010). A
blueprint for reform: The
reauthorization of the Elementary and
Secondary Education Act. Retrieved
from https://www2.ed.gov/policy/elsec/
leg/blueprint/blueprint.pdf
Vacc, N. A., & Rhyne-Winkler, M. C. (1993).
Evaluation and accountability of
counseling services: Possible
implications for a midsize school
district. The School Counselor, 40,
260-266. Retrieved from http://www.
jstor.org/stable/23901814
Vacha-Haase, T., & Thompson, B. (2004).
How to estimate and interpret various
effect sizes. Journal of Counseling
Psychology, 51, 473-481.
doi:10.1037/0022-0167.51.4.473
VanZandt, C., & Perry, N. S. (1992).
Helping the rookie school counselor: A
mentoring project. The School
Counselor, 39, 158-163. Retrieved from
http://www.jstor.org/stable/23899930
Whiston, S. C. (1996). Accountability
through action research: Research
methods for practitioners. Journal of
Counseling & Development, 74,
616-623.
doi:10.1002/j.1556-6676.1996.tb02301.x
Young, A., & Kaffenberger, C. (2011). The
beliefs and practices of school
counselors who use data to implement
comprehensive school counseling
programs. Professional School
Counseling, 15, 67-76.
doi:10.5330/PSC.n.2011-15.67
Young, A., & Kaffenberger, C. (2015).
School counseling professional
development: Assessing the use of data
to inform school counseling services.
Professional School Counseling, 19,
46-56. doi:10.5330/1096-2409-19.1.46
Earn CEUs for
reading this article. Visit
www.schoolcounselor.org
and click on Professional
Development
to learn how.