Action in Teacher Education

ISSN: 0162-6620 (Print) 2158-6098 (Online) Journal homepage: https://www.tandfonline.com/loi/uate20

Equipping Preservice Elementary Teachers for

Data Use in the Classroom

Todd D. Reeves

To cite this article: Todd D. Reeves (2017) Equipping Preservice Elementary Teachers
for Data Use in the Classroom, Action in Teacher Education, 39:4, 361-380, DOI:
10.1080/01626620.2017.1336131

To link to this article: https://doi.org/10.1080/01626620.2017.1336131

Published online: 12 Sep 2017.

Submit your article to this journal

Article views: 624

View related articles

View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=uate20
ACTION IN TEACHER EDUCATION
2017, VOL. 39, NO. 4, 361–380
https://doi.org/10.1080/01626620.2017.1336131

Equipping Preservice Elementary Teachers for Data Use in the

Classroom
Todd D. Reeves
Northern Illinois University

ABSTRACT ARTICLE HISTORY

Current preservice teacher education practice related to data use has been Received 28 February 2016
deemed inadequate, in that it is unevenly distributed and often superficial. Accepted 19 May 2017
In response, this article describes a course-based classroom assessment KEYWORDS
data-literacy experience for preservice elementary teachers. Grounded in Data-driven decision
extant theory and research concerning data literacy and data use and making; data literacy; data
interventions to promote data use processes among educators, the 6-hour use; preservice teachers;
experience involves preservice teacher scoring of teacher-developed assess- teacher education
ments, and analyzing, interpreting, and making decisions based on these
data. The pedagogical strategy’s design (objectives, materials, and activities)
and implementation process are illustrated for the reader. It is hoped that
members of the teacher education community will adopt or adapt the
strategy for implementation with preservice teachers within other programs
and institutions. In doing so, teacher educators can provide preservice
teachers with rich, in-depth experiences by which to equip them for this
salient but difficult facet of their professional practice.

In the current era of assessment-driven accountability and reform, teachers’ capacity to analyze,
interpret, and use data to inform decisions is a critical component of their professional expertise
(Mandinach & Gummer, 2013a). Professional standards require teachers to analyze and interpret
assessment data to better understand student cognition and make instructional decisions (Interstate
Teacher Assessment and Support Consortium, 2011). By informing with data decisions related to
instructional goals, methods, and time allocation, teachers can theoretically better target their
instruction to student needs, resulting in higher levels of student achievement (McDougall,
Saunders, & Goldenberg, 2007; Means, Padilla, DeBarger, & Bakia, 2009). Rigorous empirical
evidence also supports the contention that teacher engagement in data use can result in improved
student achievement growth (Carlson, Borman, & Robinson, 2011). For example, a school-wide
educator data use intervention implemented in the Netherlands was associated with overall student
achievement gains that translated to about a month of schooling (van Geel, Keuning, Visscher, &
Fox, 2016).
Research suggests that inservice teachers find the analysis, interpretation, and instructional use of
data difficult, however (DeLuca & Bellara, 2013; Stobaugh, Tassell, & Norman, 2010; Wayman &
Jimerson, 2013). This underscores the need to prepare and support teachers with respect to these
practices (Jacobs, Gregory, Hoppey, & Yendol-Hoppey, 2009; Kerr, Marsh, Ikemoto, Darilek, &
Barney, 2006; Mandinach & Gummer, 2013b). Despite the current emphasis on inservice teachers’
data literacy, recent research indicates that preservice teacher education inadequately equips teachers
by focusing only superficially on data literacy and more on assessment literacy (Greenberg & Walsh,
2012; Mandinach, Friedman, & Gummer, 2015; Mann & Simon, 2010; July). Such concerns are

CONTACT Todd D. Reeves treeves@niu.edu Educational Research and Evaluation, Northern Illinois University, 204A Gabel
Hall, 1425 W. Lincoln Highway, DeKalb, IL 60115. USA.
Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/UATE.
© 2017 Association of Teacher Educators
362 T. D. REEVES

especially salient in light of external mandates in some locales for preservice teachers to demonstrate
such skills to attain licensure. The edTPA, for example, calls for preservice teachers to engage in
processes such as analyzing, interpreting, and instructionally using data. Moreover, there is a paucity
of research concerning effective preservice mechanisms by which to equip teachers for data use
(Greenberg & Walsh; Mandinach & Gummer, 2013a; Reeves & Honig, 2015; Reeves, Summers, &
Grove, 2016).
These problems have led to calls for increased attention to the promotion of data literacy and data
use skills among preservice teachers, the future of our nation’s teaching force (Cramer, Little, &
McHatton, 2014; Data Quality Campaign, 2014). In response, the purpose of this article is to
describe a course-based classroom data literacy experience for preservice elementary teachers. The
experience engages preservice teachers in scoring (teacher-developed) traditional and performance
classroom assessments, and analyzing, interpreting, and making decisions based on the data. In
addition to describing the design of the experience, including its objectives, materials, and activities,
this article also discusses potential implementation considerations and challenges for the reader.

Theoretical Framework and Literature Review

The pedagogical strategy described herein is anchored in theoretical perspectives concerning the data
use process and data literacy. In addition, research on interventions aimed at promoting teacher data
use informed the design of the pedagogical strategy. These theoretical and empirical bodies of work
are summarized in the sections that follow.

The Nature of Data Literacy and Data Use

Data use can be understood as a complex, multifaceted, and cyclical process (Marsh, 2012). In the
context of education, Hamilton et al. (2009) define this process as “teachers, principals, and
administrators systematically collecting and analyzing various types of data . . . to guide a range of
decisions to help improve the success of students and schools” (p. 46). Teacher data literacy, “the
ability to transform information into actionable instructional knowledge and practices by collecting,
analyzing, and interpreting all types of data . . . to help determine instructional steps” (Mandinach &
Gummer, 2016, p. 2), is a related construct necessitated by data use. Whereas data use is a process,
data literacy is the ability by which one engages productively and responsibly in that process.
Teacher data literacy entails an assortment of knowledge, skills, and dispositions. Mandinach and
Gummer (2016) recently outlined five major knowledge and skill domains of this construct and their
respective facets. The first domain is identifying problems and framing questions that are to be
addressed with data, based on examination of the school context and consultation with other
stakeholders (e.g., educators, students). The second domain, using data, comprises skills such as
identifying possible sources of data, using technologies to work with data, and analyzing data
statistically vis-à-vis a problem or question. The third domain is transforming data into information,
which entails understanding how to interpret data, and different data displays and representations,
examining patterns, and articulating inferences or conclusions. Domain four, transforming informa-
tion into a decision, involves the specification of actions, such as the next instructional steps to take
and instructional adjustments to make. Finally, the fifth domain, evaluating outcomes, entails the
capacity to examine the impact of engagement in the data-use process (e.g., monitoring changes in
student performance as a function of made instructional decisions).
It should now be instructive for the reader to consider relationships between Mandinach and Gummer’s
(2016) framework and the pedagogical strategy described here. First, the present experience engages
preservice teachers in asking and answering various kinds of questions (e.g., student achievement strengths
and weaknesses, instructional implications) at multiple student levels (i.e., individual, classroom) with
classroom assessment data (identifying problems and framing questions). Second, the experience addresses
preservice teachers’ capacity to use data, specifically using technologies to work with data, and analyzing
 ACTION IN TEACHER EDUCATION 363

data statistically. Third, the experience represents the domain of transforming data into information,
including interpreting data, understanding different data displays and representations, examining patterns,
and articulating inferences and conclusions. Finally, the experience targets skills that are part of Mandinach
and Gummer’s transforming information into a decision domain (namely specifying next instructional
steps, and instructional strategies to use).
Teacher transformation of information into a decision has proved to be a particularly elusive
aspect of data use (Mandinach & Jimerson, 2016). There is evidence that, even among trained
teachers who regularly engage in data use (broadly), little time is spent considering the implications
of data for instruction, compared to other processes such as data collection and analysis (Slavit,
Nelson, & Deuel, 2013). The arduousness of this domain may be explained by its invoking of
multiple, other forms of knowledge (e.g., content, pedagogy; Mandinach & Gummer, 2016). It is for
this reason that the innovation described here culminates with opportunities for preservice teachers
to consider potential instructional implications of classroom assessment data.

Interventions to Promote Data Literacy and Data Use

As data proliferate in schools and districts, so too have interventions aimed at promoting educators’ data
literacy and engagement in data-use processes (Carlson et al., 2011; Farley-Ripple & Buttram, 2014; Marsh,
2012). However, to date these interventions have largely focused on inservice teachers. Although there is
some extant scholarship on preservice teacher engagement in action research, very few studies have focused
on preservice teacher development with respect to data literacy and use (Athanases, Bennett, &
Wahleithner, 2013; Piro & Hutchinson, 2014; Zambo & Zambo, 2007). Consequently, those looking to
implement experiences to build preservice teachers’ capacity to use data have little in the way of concrete
ideas, resources, and materials by which to guide them (Mandinach & Gummer, 2016).
Interventions implemented to promote teachers’ facility with data vary considerably in their design
(Coburn & Turner, 2011). Although to the author’s best knowledge no studies have systematically tested the
impact of particular intervention design elements, it is possible to identify design elements that were
represented in interventions for which evidence of impact is available (e.g., changes in teacher knowledge or
practice, student achievement gains). Arguably, the designs of these interventions can inform the design of
other strategies intended to promote data literacy and data use. The section that follows summarizes design
elements that have been represented in prior teacher data literacy and use interventions.1

Collaboration
Teacher collaboration is prevalent in the literature as a central element of many inservice data use
interventions (Mandinach & Jimerson, 2016; Poortman & Schildkamp, 2016; van Geel et al., 2016). At
the same time, teachers increasingly engage in collaborative activities around data in the context of data
teams and professional learning communities (Carlson et al., 2011; Hamilton et al., 2009; Jacobs et al., 2009).
Although teacher collaboration can prove challenging, it affords the pooling of teachers’ expertise that
might be necessary for difficult practices such as data use (Athanases et al., 2013; Farley-Ripple & Buttram,
2014; Means et al., 2011). A study involving preservice teacher collaboration around data also reported
pretest–-posttest increases in preservice teachers’ self-reported efficacy concerning data use practices (Piro
& Hutchinson, 2014).

Facilitation and Coaching

Previously reported and efficacious data use interventions with in- and preservice teachers have also
employed expert facilitators or coaches (Athanases et al., 2013; Roehrig, Duggar, Moats, Glover, &
Mincey, 2008; van Geel et al., 2016). In these studies, facilitators or coaches guided and assisted
teachers as they engaged in data use processes and provided targeted support and feedback. Other
work has even employed data coaching as a specific model by which to promote teacher data use,
with promising results (Marsh, McCombs, & Martorell, 2010). Relatedly, Roehrig et al. (2008)
ascribed variation in teachers’ use of progress monitoring to the unavailability of coaches.
364 T. D. REEVES

Procedures and Protocols

Another prominent feature of data use interventions is the use of clear, specific procedures and
protocols (Athanases, Wahleithner, & Bennett, 2012; Windschitl, Thompson, & Braaten, 2011). For
instance, McDougall et al. (2007) reported on a data use training that featured detailed worksheets
that outlined data-use process steps. Similarly, Gearhart and Osmundson (2009) reported on an
intervention in which science teachers learned to analyze, interpret, and use data, which was
grounded in a detailed and step-by-step protocol. Teacher inquiry scholarship has also advanced
protocols to support systematic teacher dialogue and data analysis, interpretation, and use (Nelson &
Slavit, 2008). Protocols might be particularly useful given that inservice teachers need support in
identifying questions to ask with data (Wayman & Jimerson, 2013).

Contextual Relevance
Another design element vis-à-vis data literacy and use interventions is alignment of the intervention with
teachers’ local context(s) (e.g., curriculum, assessments, students). In Gearhart and Osmundson’s (2009)
study, 23 science teachers developed data use skills through the examination of actual student work
gathered via their own assessments and from their own students. Interview-based evidence gathered
from inservice teachers also indicates that they value close alignment between data use professional
development and their classroom and curricular contexts (Wayman & Jimerson, 2013).

Use of Technology
The use of computer-based technology has also received attention in the literature with respect to the
development of teacher data literacy and data use. For example, such technology can be used to
automate data analysis processes (e.g., filtering, summarizing) and facilitate interpretation of data.
Data dashboards and data systems—which are increasingly common in schools—serve this purpose.
Technological tools used in these ways might be especially helpful given that teachers have been
shown to need support with data analysis and interpretation (Wayman & Jimerson, 2013).

Pedagogical Strategy Description and Context

The experience described here is broadly intended to promote preservice teachers’ data literacy—the ability
to analyze, interpret, and make instructional decisions based on data. Occurring during two adjacent 3-
hour weekly sessions of an assessment course, the 6-hour experience involves participants in assessment
scoring, data analysis, interpretation, and data-based decision making. As teachers can potentially use data
for myriad purposes (e.g., identifying students who need support, deciding whether to reteach), the
experience requires preservice teachers to articulate claims/decisions about overall class and individual
student performance, strengths and weaknesses, errors or misconceptions, as well as instruction.
Participants are also required to provide evidence in support of such claims/decisions. The experience
focuses on preservice teachers’ literacy related to teacher-developed assessments and classroom assessment
data specifically.
To date, the pedagogical strategy has been implemented with four cohorts of undergraduate
preservice elementary teachers at a large, public midwestern university. In this context, the
experience occurs during the last semester of preservice teachers’ coursework before student
teaching. During the assessment course in which the experience is embedded, each participant
designs and constructs a traditional and a performance assessment and corresponding scoring
guides (e.g., rubrics). A traditional assessment is an assessment in which test takers respond to a
series of questions or problems (items). In contrast, a performance assessment is an assessment in
which test takers engage in different types of (and often more authentic) behaviors or create
products (Stiggins, 1987). Although traditional assessments are often scored at the level of
individual items, performance assessments are often accompanied by rubrics wherein multiple
performance dimensions are scored.
 ACTION IN TEACHER EDUCATION 365

Preservice teachers receive in-depth feedback on their assessments from a peer and the instructor.
Participants administer their assessments to a sample of K-8 students during a clinical experience
and bring all collected student work (e.g., completed tests, projects, rubrics, checklists) to the first
session. Prior to the experience, preservice teachers also receive direct instruction in scoring student
work, summarizing item- and test-level data (tabularly, graphically, statistically), and disaggregating
data by subgroup and content standard/behavioral objective.
The design of the pedagogical strategy is grounded in theory and prior research related to data
literacy and use, especially research on successful data use interventions for pre- and inservice
teachers (e.g., Poortman & Schildkamp, 2016; Reeves & Honig, 2015; van Geel et al., 2016). Along
these lines, the experience provides opportunities for preservice teachers to engage in a variety of
behaviors inherent in data-driven decision making (Mandinach & Gummer, 2016; Wieman, 2014).
The experience addresses primarily components two through four of Mandinach and Gummer’s
(2016) framework cited earlier: using data, transforming data into information, and transforming
information into a decision. Identifying problems and framing questions is addressed only in a
limited way, as the questions to be addressed by participants are prescribed within the protocol. The
fifth component—evaluating outcomes—is not addressed due to practical constraints on preservice
teachers’ ability to reengage students vis-à-vis content instruction and evaluate the results.
The design of the experience itself is also anchored in prior scholarship on data literacy and use
interventions for teachers. First, the instructor of the assessment course (the author of this article)
facilitates the experience. Second, the experience is highly structured in nature. Materials provided to
participants include step-by-step protocols that frame each of the 2 days (see the appendix). The
instructor introduces each day by reading a script that explains the arc of the experience that day to
orient participants to each day’s scope of work. Each day of the experience also proceeds in stages,
for which a particular amount of time is allocated (e.g., interpretation of overall class performance,
interpretation of individual student performance).
Third, the assessment data the preservice teachers work with are collected from K-8 students with
whom they are completing a clinical experience. Thus, the experience itself is classroom contextua-
lized, and the participants are familiar with the students from whom data are collected. During the
experience, each preservice teacher works with his or her own traditional and performance assess-
ments, scoring guides, and corresponding data gathered within their clinical placement (even though
they may collaborate during the experience). As each preservice teacher has data from two assess-
ments to work with, participants complete each activity twice—once for their traditional assessment
and once for their performance assessment. Each preservice teacher working with data from two
assessments is intended to maximize opportunities to practice scoring, data analysis and interpreta-
tion, and data-driven decision-making processes within the experience, in turn promoting transfer of
training.
Fourth, tools are used to scaffold participants and facilitate scoring, data analysis and interpreta-
tion, and data-based decision making processes (as elaborated below). For example, an Excel
document is used to facilitate interpretation of item-level data through tabular and graphical
representation of those data. As another example, a protocol contained in a Word document serves
to structure and facilitate preservice teachers’ articulation of claims/decisions about students/instruc-
tion and the provision of evidence in support of those claims/decisions. The Excel and Word
documents are made available in electronic and hard copy format, in the event that the preservice
teachers prefer to work on paper (and in case of technology failure).
Fifth, to promote data-related collaboration, the preservice teachers are assembled in four- to five-
person grade-level roundtables approximately corresponding to the grade level of a concomitant
clinical experience. Participants previously worked in these roundtables twice during the course
(once to design and construct assessments, and once to review and revise them). Relative to
collaboration, preservice teachers are orally provided with the following direction, “While these
activities are ultimately to be completed individually, you are welcome to ask questions to and assist
your peers during any part of today’s workshop.”
366 T. D. REEVES

Day One of the Experience

Assessment Scoring
The first day of the experience is primarily focused on scoring the traditional and performance
assessments. Participants are required to bring all student work (e.g., completed tests, writing
products) or scans of such work, as well as corresponding answer keys and/or scoring guides they
developed with their assessments. The preservice teachers score their traditional assessments first
and record item scores on the students’ actual work products. For their traditional assessment, any
selected-response items are scored first, one student at a time. Then, any constructed-response item
responses are scored in accord with best practices, one item at a time and with students in random
order. Next, the preservice teachers score their performance assessments in random order (and
anonymously if possible) and record scores on the actual student work products.

Qualitative Analysis Worksheet

In general, the experience is largely focused on the derivation, analysis, interpretation, and use of
quantitative data. However, given the key pedagogical value of qualitative data as well, during the
first day participants also engage in qualitative analysis of student work. In groups, the preservice
teachers identify those assessments featuring constructed-response items that were answered incor-
rectly by K-8 students. Subsequently, the preservice teachers collaboratively and qualitatively exam-
ine students’ incorrect responses for patterns (e.g., errors or misconceptions) and provide evidence
for those patterns. Figure 1 presents an example of a completed qualitative analysis worksheet, which
provides a structure for this process. The qualitative analysis worksheet includes scaffolds related to
the nature of evidence that can be provided for particular patterns (i.e., “number of students,”
“exemplar responses”).

Raw Data Worksheets

Once all student work is scored, each preservice teacher enters the numeric scores into Excel-based raw
data worksheets, one for the traditional assessment and one for the performance assessment. In each
spreadsheet, each row represents a particular student, and each column represents a particular item or
rubric dimension. Each spreadsheet was preprogrammed by the instructor to compute total assessment
scores and descriptive statistics at the item or rubric dimension and total score levels. Each item or
rubric dimension column automatically computes descriptive statistics for the scores in that column
(i.e., item- or rubric dimension-level descriptive statistics). Preservice teachers highlight in different
colors columns related to each behavioral objective targeted by different components of the assessment
to facilitate later analysis and interpretation of the data by behavioral objective. There are also rows in
which preservice teachers indicate the maximum number of possible points for each item or rubric
dimension and provide an item or rubric dimension descriptor or key word (e.g., “punctuation”) to
facilitate later data analysis and interpretation of item- or rubric-dimension-level score statistics. The
Excel files automatically calculate total scores for each student and, after preservice teachers put an X in
the total score column for any empty rows, properly computes descriptive statistics for the total scores.
Optionally, preservice teachers can hide empty rows and columns to ease data interpretation.2 Figure 2
presents a completed raw data worksheet document for a mathematics traditional assessment.

Day Two of the Experience

The second day of the experience is primarily focused on the analysis and interpretation of the
traditional and performance assessment data and making decisions on their basis. Preservice
teachers are required to bring completed electronic copies of the raw data worksheets (e.g., emailed
to oneself, saved on a jump drive). The preservice teachers are also required to bring at least one
copy of each actual assessment itself (e.g., a completed test, assignment sheet) and corresponding
scoring guides to facilitate data interpretation and use.
 ACTION IN TEACHER EDUCATION 367

Figure 1. Exemplar qualitative analysis worksheet for incorrect counting and subtraction responses. The counting task was designed to
elicit evidence (partly) of student mastery of the following Common Core State Standard for mathematics: “Count to 100 by ones and by
tens” (K.CC.A.1). The subtraction item was intended to align partly with the following Common Core State Standard for mathematics:
“Fluently add and subtract within 1000 using strategies and algorithms based on place value, properties of operations, and/or the
relationship between addition and subtraction” (3.NBT.A.2). In this example, preservice teachers identified common counting and
subtraction errors among K-8 students. Example preservice teacher responses contained within white cells.

During this second day, the preservice teachers first engage in data analysis and interpretation
activities in accord with a step-by-step protocol (see the appendix), during which they take notes
concerning their tentative conclusions. Preservice teachers interpret measures of central tendency (i.e.,
mean, median, mode) and dispersion (i.e., range, Standard Deviation) for particular items or rubric
dimensions and total scores (all automatically calculated in the Excel document from the first day).
Preservice teachers also interpret the performance of three individual students on each assessment
overall and on particular items or rubric dimensions. In selecting individual students on which to focus,
the preservice teachers are encouraged to select students who were legitimately engaged during the
assessment process and who are members of a special population (e.g., English language learners,
students with disabilities), “average,” or otherwise interesting to the preservice teacher.
Altogether, for each of two assessments the preservice teachers practice data analysis, interpretation,
and use at different levels of student aggregation and content grain sizes. With respect to level of student
aggregation, the preservice teachers focus their efforts at the overall class and individual student levels.
In terms of grain size, the preservice teachers examine mastery of content overall, by content standard/
behavioral objective, and by item or rubric dimension. The sequence of these activities is highly
prescribed in the protocols (e.g., examination of overall class performance on the traditional assessment
overall, then overall class performance on parts of the traditional assessment, etc.). After the preservice
teachers engage in data analysis and interpretation tasks to yield tentative conclusions, they then repeat
the process to formally articulate claims/decisions about students and instruction.
368 T. D. REEVES

Traditional Assessment Raw Data Worksheet

Student Item Total

# Name 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Score
1 Camryn 2.0 2.0 2.0 2.0 1.0 0.0 1.0 2.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0 24.0
2 Addelyn 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 0.0 0.0 2.0 1.0 2.0 0.0 23.0
3 Gabrielle 2.0 1.0 0.0 2.0 2.0 1.0 0.0 0.0 1.0 0.0 1.0 1.0 0.0 1.0 0.0 12.0
4 Cadence 2.0 2.0 2.0 1.0 1.0 1.0 0.0 2.0 2.0 2.0 2.0 2.0 2.0 0.0 2.0 23.0
5 Lexi 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 0.0 2.0 28.0
6 Jaden 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 0.0 2.0 28.0
7 Mykal 2.0 2.0 2.0 0.0 0.0 2.0 2.0 2.0 2.0 0.0 2.0 1.0 2.0 0.0 2.0 21.0
8 Michael 1.0 1.0 0.0 1.0 1.0 0.0 0.0 1.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 7.0
9 Joseph 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 Yuri 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 2.0 29.0
11 Aleia 2.0 2.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 28.0
12 Hunter 2.0 2.0 2.0 2.0 2.0 1.0 1.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 27.0
13 Madison 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 30.0
14 Lydia 2.0 2.0 2.0 2.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 28.0
15 Jozlyn 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 30.0
16 Anthony 2.0 1.0 2.0 2.0 2.0 1.0 1.0 1.0 1.0 1.0 0.0 1.0 2.0 1.0 0.0 18.0
17 Rowand 2.0 2.0 1.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 28.0
18 Ava 1.0 1.0 1.0 2.0 2.0 1.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 26.0
19 Ashtyn 2.0 2.0 2.0 2.0 2.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0 2.0 0.0 2.0 26.0
20 Owen 2.0 1.0 2.0 2.0 1.0 2.0 1.0 2.0 2.0 2.0 1.0 2.0 2.0 2.0 2.0 26.0
21 Ryan 0.0 1.0 2.0 1.0 2.0 1.0 0.0 2.0 2.0 0.0 2.0 2.0 2.0 0.0 0.0 17.0
22 Blake 2.0 1.0 2.0 2.0 2.0 1.0 2.0 2.0 2.0 1.0 1.0 2.0 2.0 1.0 2.0 25.0
23 Summer 2.0 2.0 1.0 2.0 2.0 1.0 1.0 2.0 2.0 0.0 1.0 2.0 0.0 1.0 0.0 19.0
24 Tyler 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 29.0
Descriptive Statistics
Item Total
Statistic
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Score
Mean 1.75 1.63 1.63 1.63 1.67 1.33 1.29 1.75 1.83 1.25 1.50 1.67 1.58 1.08 1.42 23.00
Median 2.0 2.0 2.0 2.0 2.0 1.5 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 26.0
Mode 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 28
Range 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 30
Standard Deviation 0.60 0.56 0.70 0.70 0.62 0.75 0.84 0.60 0.47 0.92 0.76 0.62 0.76 0.86 0.91 7.44
Possible Points 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 30.00
Item Descriptor 1SA 1SS 2SAS 1SA 1SA 2SA 2SS 1SS 2SA 2SAS 2SS 2SA 1SS 2SAS 2SS -

Figure 2. Completed raw data worksheet for a mathematics traditional assessment designed to elicit evidence of student mastery
of the following Common Core State Standard for mathematics: “Use addition and subtraction within 100 to solve one and two
step word problems involving situations of adding to, taking from, putting together, taking apart, and comparing with unknowns
in all positions by using drawings and equations with a symbol for the unknown number to represent the problem” (2.OA.A.1). In
this example, the preservice teacher parsed out five separate behavioral objectives from this content standard related to (1)
solving one-step addition word problems within 100 (1SA), (2) solving two-step addition word problems within 100 (2SA), (3)
solving one-step subtraction word problems within 100 (1SS), (4) solving two-step subtraction word problems within 100 (2SS),
and (5) solving two-step addition and subtraction word problems (2SAS). In the bottom-most row of the raw data worksheet,
descriptors or key words are provided for each item to facilitate data analysis and interpretation. For example, “1SA” at the
bottom-left represents the first item (“1”) on the assessment, a two-step mathematics problem which involved first subtraction
(“S”) and then addition (“A”). The items represented within the raw data worksheet constitute constructed-response items scored
for partial credit (0, 1, or 2).

Item Score Frequency Distribution Worksheets

To assist with interpretation of the data, the preservice teachers also tabularly and graphically
represent overall class performance on select items or rubric dimensions. In particular, each
preservice teacher represents the scores for six items or rubric dimensions in the form of a frequency
distribution in Excel (three items or rubric dimensions from the traditional assessment and three
from the performance assessment). The tool used to facilitate this is termed the “item score
frequency distribution worksheet.” Intended to facilitate the preservice teachers’ interpretation of
scores from particular items or rubric dimensions through visual representation of the score
distribution, the Excel file is set up to represent graphically these data in the form of a histogram
automatically. Figure 3 presents a subset of a completed item score frequency distribution worksheet
for one dimension of a performance assessment rubric.

Data-Based Decision Making Worksheet

The culminating component of the experience involves formally making claims/decisions based on
the data. Toward this end, each preservice teacher completes a so-called data-based decision-making
worksheet. For each assessment, preservice teachers articulate in the data-based decision-making
worksheet claims/decisions about: overall class performance, strengths and weaknesses, and errors or
misconceptions; three individual students’ performances overall, strengths and weaknesses, and
 ACTION IN TEACHER EDUCATION 369

Figure 3. Example excerpt of a completed item score frequency distribution worksheet for an English/language arts performance
assessment. The particular rubric dimension represented in the figure is intended to align with the following Common Core State
Standard for English/language arts: “Distinguish among fact, opinion, and reasoned judgment in a text” (RH.6–8.8). Although the
worksheet’s title references an “item,” it is also used to represent scores derived from a particular dimension of a rubric.

errors or misconceptions; instruction (e.g., changes to the lesson plan, next steps); and feedback that
could be provided to students based on the data. For each claim/decision, preservice teachers need to
cite supporting evidence (qualitative, tabular, graphical, or statistical). The organization (sequencing)
of the data-based decision-making worksheet parallels that of the earlier second-day data analysis
and interpretation activities. As such, when completing the data-based decision making worksheet,
preservice teachers are finalizing claims/decisions they began to initially formulate when they earlier
engaged in data analysis and interpretation. Figures 4–6 contain excerpts from completed data-based
decision-making worksheets. It is notable that the data-based decision-making worksheet contains
scaffolds throughout, as shown in Figure 4 through 6. For example, in the section concerning
instructional decisions about next steps for current students, the worksheet states “(e.g., move on
to particular behavioral objectives using a particular method, reteach the behavioral objectives using
a particular method).”
Immediately prior to beginning work with the data-based decision-making worksheet, the pre-
service teachers receive direct instruction in articulating evidence-based claims about and providing
high-quality feedback to students. In terms of articulating evidence-based claims, preservice teachers
are given guidelines for stating claims and advancing evidence in support of those claims. For
example, it is explained to preservice teachers that claims should focus on students’ degree of
mastery of the content targeted by their assessments, rather than test scores (as test scores are
evidence of knowledge or skills).
With respect to the provision of evidence, preservice teachers are instructed to provide specific
and relevant evidence. In particular, preservice teachers are instructed to: indicate which specific
item(s) or rubric dimension(s) scores are the basis for each decision/claim about what students know
or can do, and explicitly state why the evidence is relevant to a specific decision/claim (e.g., because
the items required students to do X). They are also reminded to put scores in context (e.g., when
reporting a class summary score, the total number of possible points should be reported as well). In
terms of feedback, the preservice teachers are alerted to the characteristics of high-quality feedback,
such as attending to strengths and weaknesses.
Finally, before completing the data-based decision-making worksheet, preservice teachers are also
reminded to keep in mind assessment quality issues (e.g., item quality, assessment task quality,
scoring guide quality, and reliability, validity, and fairness issues) as well as students’ engagement
during the assessment process. The preservice teachers are also told that if the data do not support a
 370
T. D. REEVES

Overall Class: Traditional Assessment

Specific Decision(s)/Claim(s) Specific, Relevant Evidence (tabular,

Overall class strengths (e.g., high performance
relative to total possible points on particular
item/s, or items related to a particular behavioral
objective/content standard, knowledge/skills)
Overall class weaknesses (e.g., low
performance relative to total possible points on
particular item/s, or items related to a particular
behavioral objective/content standard,
knowledge/skills)
The class can identify what
government does at local, state, and
national levels.
The class cannot describe how local,
state, and national governments
interact.
graphical, statistical)
20/25 students (80%) answered question
1 correctly, which required students to
identify a role of local government.
22/25 students (88%) answered question
2 correctly, which required students to
identify a role of state government.
23/25 students (92%) answered question
3 correctly, which required students to
identify a role of national government.
The mean scores for questions 4, 5, and
6 were .2, .24, .32. These questions were
each worth one point and required
students to describe how different levels
of government interact.

Figure 4. Example excerpt from a data-based decision-making worksheet in which a preservice teacher articulates data-based claims about an overall class’ content mastery strengths and
weaknesses and provides evidence for those claims. The example shown is associated with a traditional assessment in a social studies context. The particular traditional assessment was designed
to elicit evidence of student mastery of two behavioral objectives related to government: (1) “Students will be able to identify what government does at local, state, and national levels,” and (2)
“Students will be able to describe how local, state, and national governments interact.”.
 ACTION IN TEACHER EDUCATION 371

Focus Students: Performance Assessment

Focus student 1 strengths (absolute and
relative)
High performance relative to total
possible points on particular item/s
High performance relative to the overall
class mean on particular item/s
Specific Decision(s)/Claim(s)
The student can state an opinion.
The student can supply reasons that
support an opinion.
The student can write a concluding
statement in an opinion piece.
Specific, Relevant Evidence (scores
received and their meaning)
The student earned 1 out of a possible 1
point, based on the part of the rubric
related to stating an opinion.
The student earned 2 out of a possible 2
points, based on the part of the rubric
related to supplying reasons that support
an opinion.

The student earned 1 out of a possible 1

Focus student 1 weaknesses (absolute and
relative)
Low performance relative to total
possible points on particular item/s
Low performance relative to the overall
class mean on particular item/s
The student cannot introduce a topic
in an opinion piece.
point, based on the part of the rubric
related to writing a concluding
statement in an opinion piece.
The student earned 0 out of a possible 1
point, based on the part of the rubric
related to introducing a topic in an
opinion piece.

Figure 5. Example excerpt from a data-based decision-making worksheet in which a preservice teacher articulates data-based
claims about an individual (focus) student’s content mastery strengths and weaknesses and provides evidence for those claims.
The example shown is associated with a performance assessment designed to elicit evidence of mastery of the Common Core
State Standard for English/language arts: “Write opinion pieces in which they introduce the topic or book they are writing about,
state an opinion, supply reasons that support the opinion, use linking words (because, and, also) to connect opinion and reasons,
and provide a concluding statement or section” (W.2.1). In this example, the preservice teacher has deconstructed the standard
and assessed the following four behavioral objectives specifically: (1) “Students will write opinion pieces in which they introduce
the topic or book they are writing about;,” (2) “Students will write opinion pieces in which they state an opinion,” (3) “Students will
write opinion pieces in which they supply reasons that support their opinion,” and (4) “Students will write a concluding statement
or section at the end of their opinion piece.”.

Focus Students: Performance Assessment

Specific Feedback
Literally write out the feedback you would give them orally or in writing (e.g., “Good topic sentence,”
“Make sure to subtract the bottom number from the top number”)
Strengths “Great opinion about your favorite animal. You did a wonderful job introducing your first and second
reasons. Your conclusion sentence wrapped up your ideas well.”

Weaknesses “Make sure to include a topic sentence at the very beginning to tell your audience what they can expect
to read about in your paper.”

Figure 6. Example excerpt from a data-based decision-making worksheet in which a preservice teacher articulates evidence-based
feedback to provide to an individual (focus) student related to their content mastery strengths and weaknesses. The example
shown is associated with the same writing performance assessment, and student, reflected in Figure 5.

particular claim/decision, they are to write, “No data.” For example, if a student answered all
multiple-choice questions correctly, it is not possible to identify or provide feedback on his or her
weaknesses related to the content assessed.

Discussion
Despite prominent attention to teacher data literacy and use, the current status of teacher education
for data use has been deemed inadequate. The treatment of data use in preservice teacher education,
in particular, is often superficial, sometimes absent entirely, and focused more on assessment and
assessment literacy than data literacy (Greenberg & Walsh, 2012; Mandinach et al., 2015; Mann &
Simon, 2010, July). At the same time, little is empirically known about how to develop such expertise
and practices during teacher education (DeLuca & Bellara, 2013; Greenberg & Walsh; Reeves &
Chiang, 2017). Addressing this concern, the purpose of this article was to describe an assessment
course-based pedagogical strategy designed to support the development of preservice teachers’
372 T. D. REEVES

assessment data literacy (i.e., ability to analyze, interpret, and make decisions based on assessment
data). The experience involved scoring traditional and performance classroom assessments, and
analyzing, interpreting, and making decisions based on the data. Grounded in theory and research
concerning data literacy and use and their development, the described experience was facilitated,
collaborative, and contextually relevant, and featured step-by-step protocols and technological tools.

Implementation Challenges
A number of notable challenges naturally occurred during implementation of the pedagogical
strategy. Some of these implementation challenges resulted in changes being made to the experience
in subsequent iterations. First, during the initial implementation of the experience it was learned that
some of the preservice teachers did not begin the experience with high-quality, ready-to-use scoring
guides. Not having a scoring guide precluded these preservice teachers’ ability to participate mean-
ingfully in the first day of the experience. In response to this challenge, the instructor decided to
formally review and provide feedback on drafts of all preservice teachers’ scoring guides prior to the
experience.
Second, some preservice teachers did not bring the collected K-12 student assessment artifacts
(e.g., completed tests, student work) to the first session; these preservice teachers were unable to
participate fully in the first day of the experience and were instructed to assist their peers during the
session. In response to this challenge, the instructor implemented additional electronic reminders to
preservice teachers concerning what they must bring to each session.
Third, the author identified problems with the evidence-based claims/decisions articulated by
participants (e.g., specificity of the claims, relevance of the evidence). In response to this challenge,
the instructor provided direct instruction during the experience concerning the formulation of
evidence-based claims and included related scaffolds on the data-based decision making worksheet.
Another implementation challenge concerns the pace of the experience. Some preservice teachers
perceived that the experience was too fast whereas others perceived that it moved too slowly (Reeves
& Honig 2015). These mixed perceptions concerning the pace potentially arose from the fact that the
participants’ assessments varied with respect to their length and scoring complexity; and, the
numbers of K-12 students to which the assessments had been administered varied. As such, the
base process of scoring, analyzing, and interpreting the data might have taken longer for some
participants. A related challenge was that that some participants moved ahead with the protocol at a
pace faster than expected, perhaps due to higher prior levels of experience using data (e.g., Athanases
et al., 2013). In response to this challenge, participants were encouraged to assist and/or discuss
conclusions with their peers if they finish prior to the group advances to the next phase.

Resource and Programmatic Considerations

Preservice teachers’ engagement in the essential features of the strategy as described here necessitates
a number of resources. First, since participants work with data gathered from actual K-12 students,
the experience as described herein necessitates access to K-12 students for data collection (e.g.,
through a partnership with a school or district). It may be possible to create a prefabricated version
of the experience with simulated data, but then it would no longer be contextualized. Second, the
experience requires preservice teacher access to computers equipped with word processing and
spreadsheet software. Third, the materials are rather expensive to print, if not disseminated electro-
nically to students. Fourth, if data are collected from actual K-12 students, it is necessary to protect
their privacy (e.g., by using identification numbers).
The experience described here also requires a considerable amount of prior knowledge on the
part of the participants, namely, the theory and practice of assessment task and scoring design. Given
the amount of prior instruction needed, the experience may be best situated in the latter portion of
an assessment course. It may also be possible to offer the experience as a stand-alone professional
 ACTION IN TEACHER EDUCATION 373

development opportunity (e.g., concurrent with a clinical experience, or as a teacher leadership

capstone), assuming participants have the requisite prior knowledge and access to classroom assess-
ment data. As the domain of transforming information into a decision necessitates pedagogical
knowledge (Coburn & Turner, 2011; Mandinach & Gummer, 2016; Young & Kim, 2010), the
experience may also be most appropriate toward the end of a teacher education program (or
pedagogical knowledge scaffolds should be provided). In sum, those considering adoption or
adaptation of the strategy should consider the prior knowledge and experiences of those preservice
teachers with whom it will be implemented.
Crucially, the pedagogical strategy occupies six in-class hours (two of fifteen 3-hour weekly class sessions
in this context), with about one half of these hours focused on assessment scoring and data entry processes.
Given this curricular burden, in the fourth of four semesters of implementation, the experience was scaled
back to only three in-class hours, and participants completed the majority of the first day’s activities (i.e.,
scoring and data entry) outside of class. Rather than directing the preservice teachers through scoring and
data entry processes in class, the instructor created a screencast instructional video to guide participants
through these processes. Participants then brought their raw data worksheets and one copy of each
assessment and its corresponding scoring guide to the in-class session. In reducing the total length of the
experience, another activity that had occurred during the first day was removed as well. Specifically, the
qualitative data analysis exercise—wherein students identified common errors and misconceptions in K-12
students’ constructed responses or performances—was removed and conducted prior to the experience
during the course. This change made the experience focus largely on quantitative (test score) data.

Concluding Thoughts and Future Directions

The pedagogical strategy described herein constitutes a needed mechanism by which to equip preservice
teachers for a salient but difficult facet of their future professional practice: data use. It is hoped that the
pedagogical strategy will be adopted or adapted by other teacher education practitioners, programs, or
institutions to support improved preservice teacher education practice. In doing so, teacher educators
can better promote preservice teacher development concerning data literacy and data use. Building the
capacity of preservice teachers to use data should in turn enhance their capacity to implement data-
driven decision making and more targeted, effective instruction in the K-12 classroom. Such experi-
ences may also better equip preservice teachers for high-stakes performance assessments such as the
edTPA and licensure attainment. Indeed, the implementation of experiences of this nature are critical
given the relatively limited attention given to data use during preservice teacher education broadly
(Mandinach & Jimerson, 2016; Mann & Simon, 2010; July). Toward these ends, all materials (e.g., Excel-
based data analysis tools, scripts) are available from the author upon request.
Future implementations of this experience could also modify its design and foci. For example, it
would be interesting to incorporate participant analysis, interpretation, and instructional use of data
at the level of K-12 student subgroups, or additional data displays and representation (e.g., box
plots). In addition, future implementations could focus on data from not only classroom assess-
ments, but also common and large-scale assessment data and nonachievement data. Alternatively,
adaptations might focus only on student data in a particular subject matter area such as mathematics
or science (e.g., as a vehicle to explore student mathematical thinking in the context of a mathe-
matics methods course).
More broadly, the strategy could be expanded to more fully represent the totality of data literacy
as defined by Mandinach and Gummer (2016). Firstly, the experience could be expanded to better
represent the domain of identifying problems and framing questions. Preservice teachers might
complete these activities in relation to a problem of relevance in a local school context (e.g., poor
achievement of a particular student population, or in a particular subject-matter area), one perhaps
identified in conjunction with practicing teachers. Secondly, another expansion would be to
374 T. D. REEVES

incorporate facets of the evaluating outcomes domain. Preservice teachers could potentially imple-
ment the instructional changes they proposed during the experience and then reexamine the status
of the problem after their implementation.

Notes
1. Studies by McDougall et al. (2007), Carlson et al. (2011), and van Geel et al. (2016) were experimental (or quasi-
experimental) in nature, whereas others cited in this section used other research designs. Although the set of
design features identified here are implicated in prior research, more randomized studies are needed to
understand unequivocally the optimal design of data use capacity-building activities for teachers.
2. In some cases, the preservice teachers’ assessments have more scored components (e.g., items), or more
students were assessed, than there are columns or rows (respectively) in the raw data worksheets; in such
cases, the instructor modifies the raw data worksheet. The instructor also offers to check any preservice
teacher’s raw data worksheets to make sure they are formulated and are analyzing data correctly.
3. If a constructed-response item is scored using an analytic rubric, score the item with respect to each rubric
dimension separately and then aggregate each rubric dimension score to yield a total score for that item.

Notes on Contributor
Todd Reeves is an assistant professor of educational research and evaluation at Northern Illinois University. His
research focuses on teacher education relative to assessment and data use.

ORCID
Todd D. Reeves http://orcid.org/0000-0001-8912-1690

References
Athanases, S. Z., Bennett, L. H., & Wahleithner, J. M. (2013). Fostering data literacy through preservice teacher inquiry
in English language arts. Teacher Educator, 48(1), 8–28. doi:10.1080/08878730.2012.740151
Athanases, S. Z., Wahleithner, J. M., & Bennett, L. H. (2012). Learning to attend to culturally and linguistically diverse
learners through teacher inquiry in teacher education. Teachers College Record, 114(7), 1–50.
Carlson, D., Borman, G., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of
data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3),
378–398. doi:10.3102/0162373711412765
Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement:
Interdisciplinary Research & Perspective, 9(4), 173–206.
Cramer, E. D., Little, M. E., & McHatton, P. A. (2014). Demystifying the data-based decision-making process. Action
in Teacher Education, 36(5/6), 389–400. doi:10.1080/01626620.2014.977690
Data Quality Campaign. (2014). Teacher data literacy: It’s about time. Washington, DC: Author.
DeLuca, C., & Bellara, A. (2013). The current state of assessment education: Aligning policy, standards, and teacher
education curriculum. Journal of Teacher Education, 64(4), 356–372. doi:10.1177/0022487113488144
Farley-Ripple, E. N., & Buttram, J. L. (2014). Developing collaborative data use through professional learning
communities: Early lessons from Delaware. Studies in Educational Evaluation, 42, 41–53. doi:10.1016/j.
stueduc.2013.09.006
Gearhart, M., & Osmundson, E. (2009). Assessment portfolios as opportunities for teacher learning. Educational
Assessment, 14(1), 1–24. doi:10.1080/10627190902816108
Greenberg, J., & Walsh, K. (2012). What teacher preparation programs teach about K-12 assessment: A review. New
York.NY: National Council on Teacher Quality.
Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using student achievement
data to support instructional decision making (NCEE 2009-4067). Washington, DC: U.S. Department of Education,
Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
 ACTION IN TEACHER EDUCATION 375

Interstate Teacher Assessment and Support Consortium. (2011). InTASC model core teaching standards. Retrieved
from http://www.ccsso.org/Resources/Publications/InTASC_Model_Core_Teaching_Standards_2011_MS_Word_
Version.html
Jacobs, J., Gregory, A., Hoppey, D., & Yendol-Hoppey, D. (2009). Data literacy: Understanding teachers’ data use in a
context of accountability and response to intervention. Action in Teacher Education, 31(3), 41–55. doi:10.1080/
01626620.2009.10463527
Kerr, K. A., Marsh, J. A., Ikemoto, G. S., Darilek, H., & Barney, H. (2006). Strategies to promote data use for
instructional improvement: Actions, outcomes, and lessons from three urban districts. American Journal of
Education, 112(4), 496–520. doi:10.1086/505057
Mandinach, E., & Gummer, E. S. (2013a). A systematic view of implementing data literacy in educator preparation.
Educational Researcher, 42(1), 30–37. doi:10.3102/0013189X12459803
Mandinach, E. B., Friedman, J. M., & Gummer, E. S. (2015). How can schools of education help to build educators’
capacity to use data: A systemic view of the issue. Teachers College Record, 117(4), 1–50.
Mandinach, E. B., & Gummer, E. S. (2013b). Building educators’ data literacy: Differing perspectives. Journal of
Educational Research & Policy Studies, 13(2), 1–5.
Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills,
knowledge, and dispositions. Teaching and Teacher Education, 60, 366–376. doi:10.1016/j.tate.2016.07.011
Mandinach, E. B., & Jimerson, J. B. (2016). Teachers learning how to use data: A synthesis of the issues and what is
known. Teaching and Teacher Education, 60, 452–457. doi:10.1016/j.tate.2016.07.009
Mann, B., & Simon, T. (2010, July). Teaching teachers to use data. Paper presented at the NCES STATS-DC 2010 Data
Conference, Bethesda, MD.
Marsh, J. A. (2012). Interventions promoting educators’ use of data: Research insights and gaps. Teachers College
Record, 114(11), 1–48.
Marsh, J. A., McCombs, J. S., & Martorell, F. (2010). How instructional coaches support data-driven decision making:
Policy implementation and effects in Florida middle schools. Educational Policy, 24(6), 872–907. doi:10.1177/
0895904809341467
McDougall, D., Saunders, W. M., & Goldenberg, C. (2007). Inside the black box of school reform: Explaining the how
and why of change at getting results schools. International Journal of Disability, Development and Education, 54(1),
51–89. doi:10.1080/10349120601149755
Means, B., Chen, E., DeBarger, A., & Padilla, C. (2011). Teachers’ ability to use data to inform instruction: Challenges
and supports. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy
Development.
Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed decision making in schools:
Teacher access, supports and use. Washington, DC: U.S. Department of Education.
Nelson, T., & Slavit, D. (2008). Supported teacher collaborative inquiry. Teacher Education Quarterly, 35(1), 99–116.
Piro, J. S., & Hutchinson, C. J. (2014). Using a data chat to teach instructional interventions: Student perceptions of
data literacy in an assessment course. The New Educator, 10(2), 95–111. doi:10.1080/1547688X.2014.898479
Poortman, C. L., & Schildkamp, K. (2016). Solving student achievement problems with a data use intervention for
teachers. Teaching and Teacher Education, 60, 425–433. doi:10.1016/j.tate.2016.06.010
Reeves, T. D., & Chiang, J. L. (2017). Building pre-service teacher capacity to use external assessment data: An
intervention study. The Teacher Educator, 52(2), 155–172.
Reeves, T. D., & Honig, S. L. (2015). A classroom assessment data literacy intervention for pre-service teachers.
Teaching and Teacher Education, 50, 90–101.
Reeves, T. D., Summers, K. H., & Grove, E. (2016). Examining the landscape of teacher learning for data use: The case
of Illinois. Cogent Education, 3(1).
Roehrig, A. D., Duggar, S. W., Moats, L., Glover, M., & Mincey, B. (2008). When teachers work to use progress
monitoring data to inform literacy instruction: Identifying potential supports and challenges. Remedial and Special
Education, 29(6), 364–382. doi:10.1177/0741932507314021
Slavit, D., Nelson, T. H., & Deuel, A. (2013). Teacher groups’ conceptions and uses of student-learning data. Journal of
Teacher Education, 64(1),8–21.
Stiggins, R. J. (1987). Design and development of performance assessments. Educational Measurement: Issues and
Practice, 6(3), 33–42. doi:10.1111/emip.1987.6.issue-3
Stobaugh, R. R., Tassell, J. L., & Norman, A. D. (2010). Improving preservice teacher preparation through the teacher
work sample: Exploring assessment and analysis of student learning. Action in Teacher Education, 32(1), 39–53.
doi:10.1080/01626620.2010.10463541
van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the effects of a school-wide data-based
decision-making intervention on student achievement growth in primary schools. American Educational Research
Journal, 53(2), 360–394. doi:10.3102/0002831216637346
Wayman, J. C., & Jimerson, J. B. (2013). Teacher needs for data-related professional learning. Studies in Educational
Evaluation, 42, 25–34. doi:10.1016/j.stueduc.2013.11.001
376 T. D. REEVES

Wieman, R. (2014). Using data to improve instruction: Different approaches for different goals. Action in Teacher
Education, 36(5/6), 546–558. doi:10.1080/01626620.2014.977755
Windschitl, M., Thompson, J., & Braaten, M. (2011). Ambitious pedagogy by novice teachers: Who benefits from tool-
supported collaborative inquiry into practice and why. Teachers College Record, 113(7), 1311–1360.
Young, V. M., & Kim, D. H. (2010). Using assessments for instructional improvement: A literature review. Education
Policy Analysis Archives, 18(19), 19. doi:10.14507/epaa.v18n19.2010
Zambo, D., & Zambo, R. (2007). Action research in an undergraduate teacher education program: What promises does
it hold? Action in Teacher Education, 28(4), 62–74. doi:10.1080/01626620.2007.10463430

Appendix
Step-by-Step Protocol (Day One)
Traditional Assessment Scoring (Individual) (60 min)

(1) Score responses to all selected-response items, using the answer key and appropriate point values (e.g., 0 = incorrect,
1 = correct) (30 min)
(a) Score selected-response items one student at a time
(b) Indicate selected-response item scores on actual student work (if possible)
(c) Indicate selected-response item scores on the Traditional Assessment Raw Data worksheet (either electronic
or hard copy)
Note: If your traditional assessment has no selected-response items, proceed to Step 2.

(2) Score responses to all constructed-response items, using the item-specific scoring guides and appropriate point
values (e.g., 0 = incorrect, 1 = partially correct, 2 = correct)3 (30 min)
(a) Score constructed-response items one item at a time (i.e., score all students on the first constructed-response
item, then score all students on the second constructed-response item, and so on)
(b) For each item, score students in random order and anonymously (if possible)
(c) Indicate constructed-response item scores on actual student work (if possible)
(d) Indicate constructed-response item scores on the Traditional Assessment Raw Data worksheet (either
electronic or hard copy)
Note: Each student’s total score will be computed automatically in the electronic version of the
Traditional Assessment Raw Data worksheet.
Qualitative Analysis of Incorrect Constructed-Responses (Collaborative) (30 min)

(3) Identify Roundtable Group members whose traditional assessments included constructed-response items
(4) Identify constructed-response items answered incorrectly by at least two students
(5) Select the two constructed-response items answered incorrectly by the largest numbers of students
(6) For the first constructed-response item
(a) Collaboratively examine incorrect responses to identify patterns (e.g., misconceptions, errors, partially
applied strategies), if any
(b) Indicate patterns and exemplar responses on the Qualitative Analysis worksheet (Item 1 section)

(7) For the second constructed-response item

(a) Collaboratively examine incorrect responses to identify patterns (e.g., misconceptions, errors, partially
applied strategies), if any
(b) Indicate patterns and exemplar responses on the Qualitative Analysis worksheet (Item 2 section)
Performance Assessment Scoring (Individual) (30 min)

(8) Score all performance assessments, using the scoring guide (e.g., checklist, rubric) and appropriate point values
(e.g., 0, 1, 2, and 3)
(a) Score students in random order and anonymously (if possible)
(b) Indicate performance assessment scores (one score per rubric dimension) on actual student work (if
possible)
(c) Indicate performance assessment scores (one score per rubric dimension) on the Performance Assessment
Raw Data worksheet (electronic or hard copy)
 ACTION IN TEACHER EDUCATION 377

Note: Each student’s total score will be computed automatically in the electronic version of the
Performance Assessment Raw Data worksheet.
Data Entry (Individual) (10 min)

(9) Enter numeric scores for each student for each traditional assessment item in the Traditional Assessment Raw Data
worksheet (electronic copy)
(10) Enter numeric scores for each student for each performance assessment rubric dimension in the Performance
Assessment Raw Data worksheet (electronic copy)
(11) Highlight item columns related to each behavioral objective in different colors in the Traditional Assessment Raw
Data worksheet (e.g., items related to behavioral objective 1 in blue, items related to behavioral objective 2 in red)
to facilitate later analysis and interpretation of the data by behavioral objective
(12) For both the traditional and performance assessment, enter in the Possible Points row for each item/task
component the maximum number of points students can receive, to facilitate later interpretation of item statistics
(13) For both the traditional and performance assessment, enter in the Item Descriptor row for each item/task
component a descriptor or key word, to facilitate data analysis and interpretation
(14) Put an X in the Total Score column for any blank rows in the Traditional Assessment Raw Data worksheet
(electronic copy)
(15) Put an X in the Total Score column for any blank rows in the Performance Assessment Raw Data worksheet
(electronic copy)
(16) Bring completed Traditional Assessment Raw Data and Performance Assessment Raw Data worksheets to next
week’s workshop

Step-by-Step Protocol (Day Two)

Data Analysis and Interpretation
When interpreting and making decisions/claims based on assessment data, consider assessment
quality issues (e.g., items/tasks, scoring guides, reliability, validity, and fairness) and students’
engagement in the assessment process. Do NOT over-interpret the data.
Overall Class: Traditional Assessment Total Score Analysis and Interpretation (10 min)

(1) Verify that the Raw Data worksheet’s total score column contains an X for any empty row
(2) Interpret the traditional assessment total score mean, median, and mode (central tendency), using the Traditional
Assessment Raw Data worksheet
(3) Interpret the range and standard deviation of traditional assessment scores (dispersion), using the Traditional
Assessment Raw Data worksheet

Overall Class: Traditional Assessment Item-Level Analysis and Interpretation (15 min)

(1) Select three traditional assessment items

(2) For each item, interpret its mean, median, and mode score (central tendency), using the Traditional Assessment
Raw Data worksheet
(3) For each item, tabularly represent the score distribution in a frequency distribution, using the Traditional
Assessment Item Score Frequency Distribution worksheet
(a) Interpret the counts in the frequency distribution
(b) Interpret the percentages in the frequency distribution
(4) For each item, graphically represent the score distribution using the Traditional Assessment Item Score Frequency
Distribution worksheet

Overall Class: Traditional Assessment Analysis and Interpretation by Behavioral Objective/Content

Standard (15 min)

(8) Examine patterns in student performance on items related to particular behavioral objectives/content standards, or
knowledge/skills
378 T. D. REEVES

Focus Students: Traditional Assessment Total Score (10 min)

(9) Verify that the Raw Data worksheet’s total score column contains an X for any empty row
(10) Select three focus students (e.g., struggling reader) who completed the traditional assessment
(11) For each focus student, interpret his/her total score relative to the mean total score for the class
(12) For each focus student, interpret his/her total score in light of the dispersion of total scores

Focus Students: Traditional Assessment Item-Level Scores (10 min)

(13) Select three focus students (e.g., struggling reader) who completed the traditional assessment
(14) Select two items
(15) For each focus student, interpret his/her item score relative to the class mean score on each of the two items

Overall Class: Performance Assessment Total Score Analysis and Interpretation (10 min)

(16) Verify that the Raw Data worksheet’s total score column contains an X for any empty row
(17) Interpret the performance assessment total score mean, median, and mode (central tendency), using the
Performance Assessment Raw Data worksheet
(18) Interpret the range and standard deviation of traditional assessment scores (dispersion), using the Performance
Assessment Raw Data worksheet

Overall Class: Performance Assessment Item-Level Analysis and Interpretation (15 min)

(19) Select three performance assessment items (rubric dimensions)

(20) For each item, interpret its mean, median, and mode score (central tendency), using the Performance Assessment
Raw Data worksheet
(21) For each item, interpret the range and standard deviation of scores, using the Performance Assessment Raw Data
worksheet
(22) For each item, tabularly represent the score distribution in a frequency distribution, using the Performance
Assessment Item Score Frequency Distribution worksheet
(a) Interpret the counts in the frequency distribution
(b) Interpret the percentages in the frequency distribution
(23) For each item, graphically represent the score distribution using the Performance Assessment Item Score
Frequency Distribution worksheet

Overall Class: Performance Assessment Analysis and Interpretation by Behavioral Objective/Content

Standard (15 min)

(24) Examine patterns in student performance for rubric dimensions related to (sets of) particular behavioral
objectives/content standards, or knowledge/skills

Focus Students: Performance Assessment Total Score (10 min)

(25) Verify that the Raw Data worksheet’s total score column contains an X for any empty row
(26) Select three focus students (e.g., struggling reader) who completed the performance assessment
(27) For each focus student, interpret his/her total score relative to the mean total score for the class
(28) For each focus student, interpret his/her total score in light of the dispersion of total scores

Focus Students: Performance Assessment Item-Level Scores (10 min)

(29) Select three focus students (e.g., struggling reader) who completed the performance assessment
(30) Select two items (rubric dimensions)
(31) For each focus student, interpret his/her item (rubric dimension) score relative to the class mean score on each of
the two items (rubric dimensions)
 ACTION IN TEACHER EDUCATION 379

Decisions
Complete the following steps and record your decisions, and evidence for those decisions, on the
Data-Based Decision Making worksheet. If the data do not support a particular decision, write “No
data.”
Overall Class: Traditional Assessment

(32) Make an overall decision about class performance on the traditional assessment
(33) Indicate any strengths and weaknesses for the whole class (e.g., high or low performance on particular items, or
items related to particular behavioral objectives/content standards, knowledge/skills)
(34) Indicate any common errors, misconceptions, partial understandings, or incompletely applied strategies
(35) Cite evidence for the above decisions (qualitative, tabular, graphical, statistical)

Focus Students: Traditional Assessment

(36) Indicate any strengths and weaknesses on the traditional assessment (high or low performance on particular
items, or items related to particular behavioral objectives/content standards, knowledge/skills) for each of three
focus students
(37) Indicate any common errors, misconceptions, partial understandings, or incompletely applied strategies exhibited
by each of three focus students
(38) Cite evidence for the above decisions (scores received and their meaning, qualitative, tabular, graphical, statistical)
Instruction: Traditional Assessment Content

(39) Based on earlier decisions/claims, indicate specific changes to the lesson (e.g., activities, representations,
resources/materials, teaching strategies), if any
(40) Based on earlier decisions/claims, indicate next steps for instruction for the whole class, if any
(41) Based on earlier decisions/claims, indicate next steps for instructions for three focus students, if any
(42) Justify any instructional decisions on the basis of theory/research

Feedback: Traditional Assessment

(43) Based on earlier decisions/claims, indicate what specific feedback you would provide about the strengths and
weaknesses of the overall class (high or low performance on particular items, or items related to particular
behavioral objectives/content standards, knowledge/skills)
(44) Based on earlier decisions/claims, indicate what specific feedback you would provide about the strengths and
weaknesses of three focus students (high or low performance on particular items, or items related to particular
behavioral objectives/content standards, knowledge/skills)

Overall Class: Performance Assessment

(45) Make an overall decision about class performance on the performance assessment
(46) Indicate any strengths and weaknesses about overall class strengths and weaknesses (e.g., high or low performance
on particular rubric dimensions, or dimensions related to particular behavioral objectives/content standards,
knowledge/skills)
(47) Cite evidence for the above decisions (qualitative, tabular, graphical, statistical)

Focus Students: Performance Assessment

(48) Indicate any strengths and weaknesses on the performance assessment (high or low performance on particular
rubric dimensions, dimensions related to particular behavioral objectives/content standards, knowledge/skills) for
each of three focus students
(49) Cite evidence for the above decisions (scores received and their meaning)

Instruction: Performance Assessment Content

(50) Based on earlier decisions/claims, indicate specific changes to the lesson (e.g., activities, representations,
resources/materials, teaching strategies), if any
380 T. D. REEVES

(51) Based on earlier decisions/claims, indicate next steps for instruction for the whole class, if any
(52) Based on earlier decisions/claims, indicate next steps for instructions for three focus students, if any
(53) Justify any instructional decisions on the basis of theory/research

Feedback: Performance Assessment

(54) Based on performance assessment data, indicate what specific feedback you would provide about the strengths
and weaknesses of the overall class (high or low performance on particular rubric dimensions, or dimensions
related to particular behavioral objectives/content standards, knowledge/skills)
(55) Based on performance assessment data, indicate what specific feedback you would provide about the strengths
and weaknesses of three focus students (high or low performance on particular rubric dimensions, or dimensions
related to particular behavioral objectives/content standards, knowledge/skills)