The Alabama Reading Initiative

Data Analysis and Evaluation of Program Outcomes 1999-2015

for the Alabama State Department of Education

Daniel Henry, Ph.D., Director, Auburn Center for Evaluation

Jonathan Carpenter, B.S.

A UBUR
CENTER FOR
E v ALUATION

RN
11. rv r s1 '
 ARI EVALUATION: ACE 2

Executive Summary

The Alabama Reading Initiative (ARI) is a statewide literacy initiative piloted in

FY1999, as a K-12 program. Since that time the ARI has gone through three dramatic
changes in its configuration. The three phases addressed in this study are Phase I:
FY1999 to FY2006, Phase II: FY2007 to FY2011, and Phase III: FY2012 to FY2015. In
Phase I, the ARI was a K-12 program. During Phase II, at the urging of then Governor
Bob Riley, State Superintendent of Education, Dr. Joe Morton, to focus ARI efforts on
the K-3 grade span and reduce support to secondary schools, made the decision. In
Phase III, beginning in FY2012, the ARI completed its expansion back to a K-12
program that was begun in FY2010. Since the survey data for this report, the ARI
has again turned its focus towards an early childhood emphasis and building-based
coaches.

During its initial phase of development the ARI succeeded in becoming a robust,
comprehensive, and effective reading reform initiative. Initially, the vision of the ARI
was to implement research-based instructional practices through the delivery of
professional development supported and sustained by full-time reading coaches
working side-by-side with educators. The deployment of the intensive, systematic
professional development was initially geared towards increasing teacher capacity
leading to improved reading proficiency of Alabama students.

The Auburn Center for Evaluation (ACE) was asked to do a data analysis of data
from two principal sources: 1. Student reading achievement scores and, 2. Surveys
administered in January 2016 to Alabama Reading Initiative teachers and coaches
by the Alabama State Department of Education (ALSDE).
 ARI EVALUATION: ACE 3

Summary of Findings:

1. Although the data show that reading achievement scores have improved for
students across the state, especially in the poorest schools, the primary goal of
achieving 100% literacy has remained unmet. However, ARMT scores ranged from a
low of 79% in 2006 to a high of 86% in 2013.

2. Schools in high relative poverty areas had students with signifi cantly lower
proficiency scores.

3. Students from low-achieving schools are receiving the most benefit from ARI.
There is an inverse correlation between literacy gains and initial literacy rates.
Students in the Black Belt of Alabama were more likely to make gains than students
in other areas of the state.

4. Though data indicate a positive impact of ARI on student achievement, data also
indicate that no one factor (staffing of ARI, funding levels ofthe program, or
curricular changes) appears to have had a singular or explanatory impact on student
achievement scores.

5. Nearly every respondent who addressed the question said that ARI was a
successful and valuable program overall and that it had a positive impact on teacher
knowledge and development, the spread of best practice in the K-12 system, and as
an initiation to new teachers in the district to a common language about teaching
reading.

6. Much of the data suggestthat a widened focus and the move from a K-3 to a K-12
coaching model has had a negative impact on teacher buy-in, support for early grade
readers, and for enthusiasm for the program among many teachers.

7. Reading coaches were more favorable about the change to the consultancy model,
with a majority indicating that the consultant role was valuable.
 ARI EV ALUATIO : ACE 4

8. One unintended consequence of the move from K-3 to K-12 coaches appears to
have been that a number of teachers without specific expertise in reading became
not only reading coaches but de-facto content experts as well, which made them
much less effective. Coaches who responded to the survey had in the majority less
than five years of experience as a reading coach.

9. Some teachers objected to ARI methodology and practice. In particular, the

changing focus from the original communities of practice, K-3 building-based focus
to a basal-reader system demanding a strict fidelity. Some teachers indicated that
the emphasis on a "fidelity to basal" system took away their ability to respond to
individual students and their enthusiasm for ARI.

10. Teachers no longer have the opportunity for two-week trainings, and many
survey respondents said that was an important factor in whether ARI was a central
part of their school's curriculum.

11. Finally as the data have shown, changes in staffing meant the original goal of
placing reading coaches focused on improving reading in K-3 classrooms has
changed several times, as have the foci and pedagogical approaches.

Recommendation:

In addition to recommending ongoing purposeful external evaluations of such a

large state program, the major findings from this report suggest that an examination
of the entirety of ARI, perhaps by a blue ribbon panel of experts in the field of
reading. It should be evaluated in regard to its fidelity to the original mission of the
program to issues of how professional development is delivered, the extent to which
teachers and children in Alabama schools are being supported, and the extent to
which the attendant costs of the program are in line with its benefits to the children
of Alabama.
 ARI EVALUATION:ACE 5

Table of Contents

Executive Summary 2

Summary of Findings 3

List of Figures 6

List of Tables 7

Data Sources 8

Limitations 9

Background 11

Funding 13

Goals and Focus of Evaluation 17

Achievement Data Results 19

Survey Data Results 26

Black Belt Case Study 34

Current ARI Direction 37

Summary of Findings 38

References 41
 ARl EVALUATION: ACE 6

List of Figures

Figure 1. Alabama ARMT scores by year. 20

Figure 2. Alabama ARMT scores by grade level. 22

Figure 3. Literacy rate (NAEP) by year. 23

Figure 4. The Black Belt Region ofAlabama (from Wikipedia). 37

Figure 5. Alabama literacy rate (ARMT) between Alabama and the Black Belt
Region. 38
 ARI EVALUATION: ACE 7

List of Tables

Table 1. The ARI Growth, by Year and Cohort 11

Table 2. ARI appropriations and staffing 1999-2015 15

 ARIEVALUATION: ACE 8

Data Sources

An ex-post facto collection of data, both in the form of historical achievement

reading data of different measures, including the Alabama Reading and Mathematics
Test (ARMT), and the National Assessment of Educational Progress (NAEP) was
provided to the evaluation team. In addition, two electronic survey instruments that
were opened for teacher and reading coach responses from January 1-15, 2016
were provided. The surveys were entitled ARI Program Evaluation Teacher Survey
and the other was the ARI Program Evaluation Coaches Survey. These data were
provided, in the case of student achievement by the Alabama State Department of
Education's research division. Survey data were made available on the Survey
Monkey site on which they were captured, and made completely available to the
evaluation team. Although these data provide a great deal of information about
current and historical trends in program achievement, they are by their nature,
limited. For example, the first evaluation of the project (Moscovich, 2001) used site
visits and focus groups to evaluate the project as well as statistical analyses to gain a
deeper understanding how successful the project's launch had been. A deeper
context and the ability to ask more focused questions would have made the findings
more accurate.

Historical Overview of Statewide Reading Assessments

The test used to measure literacy in Alabama has undergone numerous changes
during the ARI's existence. The ARI was developed in part to address persistently
low reading proficiency levels on the SAT 9, a norm-referenced assessment. The
nationally-normed SAT 9 was superseded by the SAT 10 in 2002-2003. For the
purposes of this paper the SAT 9 and SAT 10 will be referred to simply as the SAT. In
2003-2004, a criterion-referenced test called the Alabama Reading and Mathematics
Test (ARMT) was piloted in Grades 4, 6 and 8. Beginning in 2004-2005, and
continuing through 2012-2013, the ARMT was administered statewide to all
students in Grades 3 through 8. The ARMT was replaced in 2013-2014 with the ACT
 ARIEVALUATION: ACE 9

battery of norm-referenced assessments, specifically, the ACT Aspire given in

Grades 3 through 8, the ACT Plan that was administered in GradelO, and the ACT
given in Grade 11. The only test that has been in continuous use since the ARI's
inception is the National Assessment of Educational Progress (NAEP).

Limitations of the Evaluation

Listed below are areas of potential weakness or limitations one should consider
when interpreting the results ofthis evaluation:

1. Data for the evaluation were completely dependent on the quality and quantity of
information collected and reported by the ALSDE and Survey Monkey.
2. Transience or mobility of the student population in participating schools is a
concern. Comparisons made in the evaluation are based on the assumption that
children in participating schools received the "treatment" of the ARI, but a more
extensive per-pupil analysis of student exposure to ARI and student mobility is
necessary to fully understand this factor.
3. Intermediate and long-term shifts in knowledge, attitudes, perceptions, and
achievements in reading may not have resulted from the ARI alone. Many schools in
Alabama have multiple federal and state initiatives in effect at the same time (e.g.
AMSTI, 21st Century Community Learning Centers, etc.). Since many ifthese
initiatives are aimed at providing increased academic achievement, this may or may
not have had an impact on ARI test scores.
4. Interpretations based on statistical significance alone should be made with
caution.
5. Although the text box entries on the survey forms produced thousands of written
responses to questions about how teachers had experienced ARI those responses
represent a small minority of teachers: there was approximately, across surveys, a
less than ten percent response rate to the written questions compared to the check-
off responses on the survey. That is, less than one in ten teacher respondents who
answered demographic questions answered with further written feedback. It is
possible that those who took the extra time to answer may not be completely
 ARI EVALUATION: ACE 10

representative of how many, or even most, other teachers in the program

experienced it.
6. This study was limited by the unavailability of consistent, annual measures of
literacy. Data provided from the ARMT were sufficient to examine relationships
during the years utilized but were nonequivalent to other measures. As such,
making valid claims regarding changes between measures is difficult.
7. There were many variables that changed across the span of the study that cannot
be accounted for by quantitative measures. Some of these include changes to
teacher training, numbers of students exposed to ARI methods, continuation of ARI
methods implementation by teachers beyond the reported date, etc.
 ARI EVALUATION: ACE 11

Background: A Brief History of the Alabama Reading Initiative

ARI Phase I Implementation*

The ARI Phase I implementation spans the FY1999 to FY2006. The focus ofPhase I
included three points of emphasis: beginning reading, expansion of reading power, and
effective intervention. In its first year of implementation, approximately 625 teachers,
administrators, counselors, librarians, and central office staff participated in initial
training to support literacy efforts for 8,662 students in 16 demonstration sites. From
FY1999 to FY2006, 900 schools were added to ARI. Schools were added to ARI by
increments as cohorts every year. Table 1 shows the ARI growth by year, cohort and
number of schools. (Oranika, 20 16).

*Note: This section contains historical information provided by ALSDE, some of which is used verbatim
from ALSDE documents, to provide background about the program. This report owes a great debt to the
assistance of the Office of Re earch and Development and in particular Patience Oranika, Ed.D., who not
only assisted the evaluation team in getting access to the needed data for analysis in an extremely timely
manner, but also provided an excellent internal evaluation ofARl which served as a source for crucial
historical and structural information for this report.

Table 1. The ARl Growth, by year and cohort

1998-1999 Cohort 1 16 Schools

1999-2000 Cohort 2 81 Schools

2000-2001 Cohort 3 267 Schools

2001-2002 Cohort 4 423 Schools

2002-2003 Cohort 5 450 Schools

2003-2004 Cohort 6 485 Schools

2004-2005 Cohort 7 511 Schools

2005-2006* Cohort 8 753 Schools

2006-2007 Cohort 9 900 Schools

*Note. All K-3 Schools statewide were trained by the end of FY2006

A breakdown of the total number of ARI schools at the end of Phase I showed: 79
middle, 86 high schools, and 735 elementary schools. During Phase I, Alabama was one
of three states to receive a federal Reading First grant with an initial allocation of
 AR1 EVALUATION: ACE 12

$16,105,592 in 2002. In 2003 , parallel to the work to the ARI, the Alabama Reading First
Initiative (ARFI) began. Through this initiative low-performing Alabama schools with
grades K -3 competed for sub-grants of $200,000 or more to support research-based
reading programs, substantial professional learning, and coaching for teachers to improve
student outcomes. In 2003-2004 74 schools were named ARFI schools. The ARFI federal
allocations continued from FY 2003 through FY 2009, and the total number of ARFI
schools grew to 93. Two prominent requirements of ARFI schools included the
purchasing of a scientifically based comprehensive reading program, from a short list of
ALSDE approved programs, and the hiring of a full time reading coach. (Oranika, 2016).

ARI Phase II Implementation

ARI Phase II implementation began in FY2007 and continued through FY2011. In 2007,
Governor Bob Riley and the state legislature provided funding to support a reading
program for every K-3 school in the state. With that commitment from the governor and
the legislature, Dr. Joe Morton, then State Superintendent of Education, announced in a
September 18, 2007 memo "All state-funded ARI reading coaches should contain their
efforts in Grades K-3 ." The ARI had been in existence for nine years when 2007 National
Assessment of Educational Progress (NAEP) revealed that Alabama students in Grade 4
made more gains than any other state in the history of the assessment.

During Phase II, a secondary pilot project involving 14 schools with combinations of
Grades 4-9 began. The name of the pilot was the Alabama Reading Initiative Project for
Adolescent Literacy (ARI-PAL). The ARI-PAL schools were all locally funded and they
committed to implementing a whole school reform model, including a full-time literacy
coach and an intensive intervention program for struggling readers. The ARI-PAL did not
expand in its second year of implementation. According to the American Institute for
Research (2006) , this was due in part to a recognition of the need to provide
additional training and professional development focused on integrating and
implementing reading across all content area courses.

The second cohort oflocally funded ARI-PAL schools was added in 2008-2009.The ARI
secondary regional coaches provided weekly support to Cohort 1 and Cohort 2 ARI-PAL
 ARI EVALUATION: ACE 13

schools. A third cohort consisting of low-performing schools that qualified for the federal
School Improvement Grant (SIG) funds was added in 2009-2010. The SIG specifications
included contracts between schools and external professional development providers
which limited the extent of ARI support in SIG schools. Additionally, the ARI state and
regional staff were divided into 11 regional teams corresponding with the 11 regional in-
service centers. The ARI state staff members became team leaders and coordinated the K-
12literacy work in the regions. The primary delivery of professional development geared
toward school leadership teams was titled, " Literacy and Justice for ALL." (Oranika,
2016).

ARI Implementation Phase III

ARI Phase III implementation spans FY2012 to present. Phase III introduced changes to
the role and focus of ARI which was modified to align with the new goals of the ALSDE.
ALSDE introduced the concept of Regional Planning Teams (RPT) and Regional Support
Teams (RSS). The goal was to provide a "one voice" support approach from the ALSDE
to the local level. Although, ALSDE provides differentiated support to all public schools
in the state of Alabama, the level and intensity of that support is currently dependent on
the unique needs as determined and requested by each school. Thus, the support structure
of ARI involves collaboration with representatives from other sections within the
department. ALSDE also adopted the use of ACT battery of assessments for Grades 3 to
11 This phase essentially ended the previous practice requiring the use of the DIBELS for
monitoring K-3 reading growth. (Oranika, 2016).

Historical Funding Changes Throughout ARI:

In FY1999, the ARI was not funded , but the initiative was present in 16literacy
sites. The LEAs paid for their reading specialists.
By FY2000, the ARI budget was $6 million. There were 81 ARI schools with
funding for 81 reading specialists in the literacy demonstration sites. Regional
coach trainers provided statewide professional development as support for literacy
demonstration sites.
 ARl EVALUATIO : ACE 14

From FY2001 to FY2004, the ARI remained a K-12 initiative and the ARI budget
increased yearly as new schools joined the initiative. By FY 2003 , the ARI budget
was $12.5 million and there were 450 ARI schools. The commitment agreement
from LEAs and schools remained in place. The state funding provided one
reading specialist for each participating LEA, with incentives to LEA's with
multiple participating schools. The ARI budget increased further from $40 million
in FY 2005 to fund 511 ARI schools to $55 million to fund 752 ARI schools by
FY2006.
FY2007, as the number of trained ARI schools grew, the ARI budget was fifty-six
million dollars. There were 899 ARI schools (all K-3 schools), however, funding
for hiring school coaches dropped from $64,167 to $55,000 per school. The
school-based coaches were supported by 96 Regional Principal/Reading coaches
The ARI budget fluctuated from over sixty-four million dollars in FY 2008 to
$59,609,850 by the close ofFY 2010. There were also fluctuations in the numbers
of school based coaches, Regional Staff, operational structure and scope and
funding allocations for hiring school based reading coaches.
The ARI budget for FY 2009 dropped slightly from $66,581million the previous
year to sixty million dollars with proration.
In FY 2010, the state budget was reduced further to $59,609,850 due to proration.
The allocation for hiring school-based coaches was once again cut from $65,581
to $61 ,587 per school. Regional Coach Staff was cut from 90 to 80. The structure
of ARI was again changed and support went to back to K- I2. This was the last
year for concentrated support for K-3 and ARI-PAL.
The ARI budget was .cut from $65 million in FY20 II , down to $48,2I2,043 in
FY2015 . There were 920 ARI schools, 788 of them were K-3 schools. Schools
continued to receive $61 ,587 per coach allocation. LEAs were asked to develop a
plan for possible expansion of coaches beyond third grade per State
Superintendent's directive. The number of Regional Coaches dropped from 80 in
Phase II to 55 in Phase III. ARI was asked to expand its roles and responsibilities
from a 100% K-3 literacy focus in 920 schools to a K-12 literacy in more than
1400 schools. In addition, reading coach support was no longer restricted to
 ARI EVALUATION: ACE 15

reading teachers and reading classrooms. Many reading coaches were reclassified
as instructional coaches and expected to support other content area
classrooms/courses. Table 2 is a graphic representation of ARI budgets from
FY2000 to FY2005:

ALABAMA READING INITIATIVE FUNDING

Table 2. ARl appropriations and staffing 1999-2015

YEAR STATE APPROJRL\TIO~ ARI SCHOOLS ARFI FEDERAL ARI STAFF

I<T!\ii>S

FY 1999 $0 16ARI 1 State Staff

(Received $1 ,500,000 from

private funding)

FY 2000 $ 6,000,000 81 ARI 3 State Staff

FY 2001 $10,240,000 267 ARI 5 State Staff

FY 2002 $11,300,000 423 ARI 11 Regional

FY 2003 $12,500,000 450 ARI $16,105,592 26 Regional

FY 2004 $12,500,000 485 ARI $18,082,502 31 Regional

74 ARFI Schools

FY 2005 $40,000,000 511 ARI $19,040,035 40 Regional

75 ARFI Schools

FY 2006 $55,000,000 752 ARI $19,373,142

93 ARFI Schools

FY 2007 $56,000,000 899 (all K-3) ARI $19,301,943 103 Regional

94 ARFI Schools

FY 2008 $64,404,704 902 (all K-3) ARI $17,969,131

95 ARFI Schools

FY 2009 $69,443,081 914 (793 K-3) ARI $ 7,308,254

 ARI EVALUATION: ACE 16

93 ARFI Schools

FY 2009 $61 ,804,342 914 (793 K-3) ARI $ 7,308,254

Prorated 93 ARFI Schools

(11 % )

FY 2010 $64,443,081 920 (788 K-3) ARI $2,000,000

(carry forward,
approximate)

FY 2010 $59,609,850 920 (788 K-3) ARI $2,000,000

(carry forward,
Prorated approximate)
(7.5% )

FY 2011 $59,952,360

FY 2012 $56,847,257 Current ARI Staff

State Staff
FY 2013 $58,458,789
1 Program coordinator
FY 2014 $48,153,789
3 Administrators
FY 2015 $48,212,043
11 Specialists

3 Clericals

68 Regional Staff

$45 million to LEAs for approximately 780 local coaches

 ARI EVALUATIO : ACE 17

Goals and Focus of the Evaluation:

The main focus of any program evaluation is to provide information to program
managers about specific program aspects in order for them to make decisions
informed by data about future directions of the program. Mostly commonly these
program aspects involve answering this question: For whom, under what conditions.
and to what extent does the program achieve its stated goals?

In the case of the founding goals of ARI there were two main objectives: Achieving a
literacy outcome goal of 100%, and second, establishing the support that is
necessary to achieve the outcome goal in the form of staffing, support, funding, and
professional development is in place. In order to understanding the current state of
the program these questions must be answered:

1. What is the current state of progress towards the outcome goal of 100%
literacy?
2. What is the current (and historical) state of the ARI in providing support for
achieving the outcome goal, and what barriers both current and historical
exist in reaching the intended goals?

In order to understand these major questions several sub-questions were

used to focus the evaluation more narrowly:

Achievement Data Questions:

1. What progress has been made toward the goal of 100% literacy?

2. What factors predict student success regarding proficiency in literacy?

3. Which students are benefitting the greatest from the implementation of ARI?

4. What grade-level differences are associated with ARI?

5. Are differences in achievement associated with the caseload of instructors?

 ARI EVALUATION: ACE 18

6. Are differences in achievement associated with state funding of ARI?

7. Are differences in achievement associated with curricular changes?

Survey Data Questions:

8. What do the data show are common experiences with the program and what are
current perceptions about ARI for the teachers and coaches who answered the
survey?

9. How do the teachers and coaches see changes made to ARI over time, particularly
in the focus of coaching and pedagogical emphasis and how have those changes
affected their practice and enthusiasm for the project?
 ARI EVALUATION: ACE 19

Achievement Data Analysis and Results

In total, data were collected annually for all students, grades 3rd- 8th, across the
years FY 2006 through FY 2016. During this time, Alabama Reading and Math Tests
(ARMT) were used as common assessment tools before changing to the ACT Aspire
the remaining three years. As such, these analyses will rely primarily on the ARMT.
During these years, over 4 million literacy scores were generated. The following
analyses will examine these literacy scores (1) to determine the level of progress
toward the goal of 100% literacy, (2) as they relate to per-pupil expenditures from
the ARI, and (3) to examine potential differences in scores between the state as a
whole and the local education agencies (LEAs) considered to be part of the Black
Belt region of Alabama. Additionally, scores from the ARMT and National
Assessment of Educational Progress (NAEP) regarding the percent of students at or
above proficient will be examined as a means of determining progress toward 100%
literacy as it relates to the implementation of ARI and the level of funding provided.

Data were provided from the Alabama State Department of Education (ALSDE) with
regard to the Alabama Reading Initiative (ARI) funding and implementation,
demographic data for students involved in this initiative, and literacy scores for
these students from the years FY 2006 through FY 2016. Initial data retrieval
involved 11 separate spreadsheets corresponding to the years FY 2006 through FY
2016 as well as a separate spreadsheet listing the level of funding provided to each
LEA during these years and another separate spreadsheet listing NAEP scores.

Due to the volume of data, it was not possible to create a single, all-encompassing
dataset. As such, many individual datasets were created for each specific purpose.
They inc! uded cleaned versions of each of the 11 annual reports with funding levels
and per-pupil expenditures added as variables. Funding was determined by
providing the raw dollar amount per LEA from the spreadsheet provided. Per-pupil
expenditures were determined by dividing the funding for each LEA by the number
of students within that LEA for each year. Additional datasets were created to
examine grade-level differences. The same variables were created and added, this
 ARIEVALUATIO : ACE 20

time to six datasets, one for each grade 3rd- 8th. For the total score assessment, a
dataset was generated from the aggregated scores for each LEA

Literacy was defined as scoring at or above the level of "proficient" on the ARMT,
which is a score of 3 or 4 on a scale of 1 - 4. These scores were dichotomously
recoded to reflect the percent of students who are proficiently literate.

Analyses

The quantitative section of analysis involves examining the progress in literacy

toward the goal of 100%, examining the relationship between ARI and literacy, and
examining any differences between the Black Belt region and the rest of the state.
These overarching purposes will be achieved through the examination of seven
related questions.

1. What progress has been made toward the goal of 100% literacy?

The primary goal of achieving 100% literacy has remained unmet. ARMT scores
ranged from a low of 79% in 2006 to a high of 86% in 2013. One-way AN OVA
demonstrates that this is a significant, positive difference (FcaJ = 4989.41, p <
.001).

Figure 1. Alabama ARMT scores by year.

Literacy Rate (ARMT) by Year

90% .----------------------------------------------

80% +-~~~~~--~~~~::::~::::!:=------------

70% +----------------------------------------------

60% +----------------------------------------------

50% +-----r-----r----.-----.-----.-----.-----r----,
2006 2007 2008 2009 2010 2011 2012 2013
 ARI EVALUATION: ACE 21

2. What factors predict student success regarding proficiency in literacy?

Schools in high relative poverty areas (LEAs with greater than average rates of
free/reduced lunch) had students with significantly lower proficiency scores (r =-
.11, p < .001) .

3.Which students are benefitting the greatest from the implementation of ARI?

Students from low-achieving schools are receiving the most benefit from ARI. There
is an inverse correlation between literacy gains and initial literacy rates (r =-.24, p <
.001). Calculations were made by subtracting 2006literacy scores from 2013
literacy scores (gains) and correlating them with the 2006literacy scores. Thus,
schools having the lowest literacy rates early on demonstrated the greatest
improvement across the study.

4. What grade-level differences are associated with ARI?

All grades have shown improvement, however, this graph demonstrates the long-
term effects of ARI by using 8th grade as a de facto control comparison. Students in
3rct grade in 2006 began receiving ARI in 2002. They were also in 8th grade in 2011,
which is the inflection point in the graph of 8th grade literacy. These students
represent the first cohort with the majority receiving directed ARI instruction and
show a substantial improvement over the previous 8th grade cohorts who had not
received direct ARI instruction.
 ARI EVALUATION: ACE 22

Figure 2. Alabama ARMT scores by year and grade level.

Annual Literacy Rate (ARMT) by

Grade Level
90%

80%

70%

60%

50%
2006 2007 2008 2009 2010 2011 2012
~ 3rd - 4th -+- 5th ~ 6th - . - 7th - 8th

5. Are differences in achievement associated with the case load of instructors?

Although the program started small, it eventually grew to serve over 300,000
students. However, the student population being served remained consistent across
the last several years. Data regarding the number and method of implementation of
ARI staff per student body were unavailable to answer this question accurately.

6. Are differences in achievement associated with state funding of ARI?

There is no statistically significant relationship between per-pupil expenditures and

proficiency scores. It is of note that funding levels increased during early years and
decreased sharply in later years. However, differences in literacy scores between
years remained consistently positive.

7. Are differences in achievement associated with curricular changes?

The only metric for measuring literacy across all curricular methods is the NAEP. As
it is only reported via total state scores, it is impossible to conduct robust statistical
analysis. However, a simple view of the percent of students achieving proficiency at
4th and 8th grade each year appears to illustrate a two-part relationship. Students
who were given directed ARI instruction in schools with K-3 coaches continued to
 ARI EVALUATION: ACE 23

show improvement after instruction ended (2003 /2007, 2005/2009, 2007/2011).

Students who received instruction in schools with K-3 coaches early on but also received
support from a K-12 ARI coach in later grades also continued to show improvement
(2009/2013 , 2011 /2015). However, students in schools only receiving instruction in
lower grades through the K-12 ARI coaching model were the first two classes to repor:t
lowered literacy rates than previous years (2013 & 2015, 4 th grade).

Figure 3. Literacy rate (NAEP) by year.

Biannual Literacy Rate (NAEP)

between 4th and 8th Grade
90%

80%

70%

60%

50%
2003 2005 2007 2009 2011 2013 2015

Suggestions for Further Analysis

It woul d be beneficial to follow class cohorts across years in order to examine the
changes experienced by each cohort of students as opposed to viewing changes
associated with years, as done in this and previous reports. Growth curve analysis
could isolate specific factors associated with ARI or other initiatives to better
answer the question of how to achieve 100% literacy.

By whatever method, it is recommended that this study receive ongoing evaluation

in order to ascertain additional differences associated with the most recent changes
to ARI as well as any future adjustments.
 ARI EV ALUATIO : ACE 24

Survey Data Analysis and Results

In all, 5241 ARI teachers and 690 coaches answered the ARI surveys made available
by the ALSDE in January 2016. In the present evaluation, the qualitative data
generated by the survey questions were analyzed using Grounded Theory. The
central methodological component of grounded theory is the method of constant
comparison (Glaser & Strauss, 1967). In this method, theories and hypotheses (or
propositions, as Pandit (1996) calls them) begin to be generated as the researcher
works with the data. The researcher begins to take notes or generate memos about
the connections and emerging patterns that are observed. This method generates
categories of data that can be confirmed or disconfirmed by subsequent data.

Glaser and Strauss (196 7) say one of the most important aspects of theory building
is the specification of concepts. In their view, hypotheses are the building blocks
between an idea the researcher has and the generation of theory. As such, they do
not bear the burden of traditional hypotheses, i.e., the need to be proven. As they
say, "generating hypotheses requires only enough evidence to establish a
suggestion-not an overwhelming piling up of evidence to establish a proof' (p.40).
For this reason, Pandit (1996) advocates the use of the word "proposition" instead
of "hypothesis" because the former is more suited to the realm of concept
development. Glaser and Strauss put a premium on emergent theory that comes
from the data and is contained in categories that demonstrate, through constant
comparison, they are most relevant and best suited to explain the data. Above all,
grounded theory allows for flexibility of methods that permit the theoretical
elements within the data to best emerge. As Dick (n.d.) comments, the "distinction
between 'emergence and forcing', as Glaser frames it, is fundamental to
understanding the method. What most differentiates grounded theory from much
other research is that it is explicitly emergent. It does not test a hypothesis" (p. 4) .

After importing the data from Survey Monkey, individual response were coded in
the software program Atlas.ti and a codebook was developed as themes emerged
 ARI EVALUATION: ACE 25

from the data. Through the iterative process of constant comparison initial codes
were developed to help understand categorize the data. Examples of some of the
themes found in the initial coding pass include "Negative about shift to Consultants"
and "Positive About ARI Coaching". As themes achieved saturation, that is, as no new
themes were found in quotations coded, saturation had been achieved. As themes
emerged across responses and surveys, larger and broader themes, reflected in the
findings below. Finally, a third, axial coding pass (see Pandit, 1996) was again
undertaken to look for confirmatory evidence of developing findings.

In the data analysis and presentation below, findings are followed by

exemplar quotes from the qualitative data illustrating and grounding in data
the findings.

NOTE: Survey responses are presented in italics and in a smaller font to

differentiate them from the analysis. Quotes are presented verbatim with identifying
characteristics (such as specific schools or coaches) removed and replaced with
XXX.

8. What do the data show are common experiences with the program and what are
current perceptions about ARI for the teachers and coaches who answered the
survey?

Findings:

1. Nearly every respondent who addressed the question said that ARI was a
successful and valuable program overall and that it had a positive impact on
teacher knowledge and development, the spread of best practice and in the
K-12 system, and as an initiation to new teachers in the district to a common
language about teaching reading.
 ARI EVALUATION: ACE 26

21 When the began teaching in 2007, I attended a week long training that was just for
hndergarten teachers. This was extremely useful because I knew how to progress monitor
correctly. I was even able to come back and train the teachers who was not able to attend.
65 Several summers ago I and another teacher went to XXX to an AR1 training. We got a hotel
and sent three days. It was the one of the best training I have been to and I am happy that we
treated ourselves to this meaningful experience.
66 ARI training was very informative and interesting when I took it. I would like refresher training
330 I truly became a "Reading Teacher" after going thru AR1 training
78 Reading coach- GREAT! always provides us with everything and if she doesnt know or doesnt
have it, she will find it
104 I am a special education Language Imparied teacher that has a self-contain classroom with
student with IQ's with 55 and below. Some have limited speech and can barely read. This year for
the first time I have had a Reading Couch come to my classroom to give me materials and I was
able to collorabated with her on some ideals on to help increace the reading for the student that
can read. Then I do not see her anymore because she now has to go and support two different
schools. We have had some very good reading couches, but Mrs.XXX has gone the extra mile to
help me.
106 She is always their to help in and out of the classroom with the students and assist me if
needed.
318 Reading Coach worked closely with me to plan whole group and small group instruction 1/4/2
316 She helped me in any areas i was lacking in and gave me ideas to better my teaching in those
areas.
313 My building coach supported me through coaching cycles, lesson planning, mahng stations
work in my classroom, and daily questions I have about standards based instruction
309 My reading coach at the time of my training had a wealth of knowledge from her many years
of experience. She was very helpful.
304 I am extremely grateful for the AR1 Coaching community. The feedback, wisdom, and
professional support ha afforded me the opportunity to work with students at various and
multiple levels of learning development in reading.
302 The reading coach was very helpful
481 Our reading coach (instructional coach) is still available to anyone who needs her. She is a
great resource and has an outstanding number of materials to help our students become great
readers.
295 She modeled lessons, answered questions and would help with small reading groups. This was
very beneficial.
107 Our reading coach is wonderful. If we need any help in anyway, she is willing to go above
and beyond!
111 My reading coaches did a great job ofanswering questions and assisting me in the classroom.
119 Our reading coach was very supportive of me and implemented AR1 strategies
120 My reading coach is great at providing support when I need it as well as when I do not. She is
awesome!!!
123 Our reading coach usually tried to make herselfavailable as needed. I suppose it averaged
out to be weekly or biweekly.
291 Our reading coach is great and is always there willing and able to help all the teachers in our
school!
290 Support from my reading coach is great!
 ARI EVALUATION: ACE 27

2. In addition, data, both qualitative and quantitative, suggest that achievement

relative to the rest of the nation as well as previous years seems to have plateaued.
Data from all cohorts in answering the survey question of how well ARI supports
them responded that ARI not only has lost much of the original "buzz" or
enthusiasm it originally had, but it also has been broadened in its approach to the
extent that it has become ineffective in many of its core missions. Many teachers
who took the survey were surprised that ARI was still an extant program:

27 This role has morphed and changed through the years. At our building the role is that of a
dyslexic service provider, progress monitoring administrator, compiler ofdata, assistant to the
principal.
23 Our focus has shifted from AR1 to math over time
864 Gradually the emphasis lessened on the components and moved to more of my reading
coach's agenda.
76 We did receive lots of support but since our system has gone to grade level coaches they are
now at our school every third week and it seems that when they are here or needed they schedule
meetings at the county office for them to attend.
10 This is the first time I have heard the term ARI mentioned in YEA RS!!!
118 Back in 2006 we received a lot of support, but now, we receive very little. No one mentions
the strategies that were used in AR1 anymore.
121 transferred to high school in 2007 and until the last couple ofyears that support has been
nonexistent
226 When we were first trained, and for several years post training, we always had PD's or
visitors from ARI. I have seen none of that is a really long lime.
310 Honestly, it's not an acronym used often these days!!!
314 I wasn't even sure we still did AR1 because we haven't received support in so many years. We
have had recent school discussions abouting how to train non-veteran teachers about the use of
ARI strategies. I would very much like for there to be a renewal. I am a believer in the
effectiveness ofARI. I have noticed that new teachers are not familiar with the initiative.
544 It seems to come and go. We were heavily into it then we hear nothing about it.
I 0 I think it's a great program from what I remember about it. I also believe we as a district, need
to stick to one proven success program. Results won't manifest in a year. We can keep flip flopping
from Wonders to ARI. Just my two cents.

9. How do teachers and coaches perceive changes made to ARI over time, particularly in
the focus of coaching and pedagogical emphasis and how have those changes affected
their practice and enthusiasm for the project?

Findings:

1. Much of the data suggests that a widened focus and a K-1 2 vs. K-3 model has had a

negative impact on teacher buy-in, support for early grade readers, and for enthusiasm for
the program among many teachers:
 ARl EVALUATION: ACE 28

144 The coaches are spread thin. Often times they service more than one school. They
have a lot of other roles that often take them away from being able to help teachers
frequently. ARI related to special education students .... no materials. What additional
materials are there to help our students succeed in reading. Programs? Online
programs? Etc
143 The support role was very helpful for several years. The reading coach was someone
who would offer advice and new strategies to help struggling readers. Now, the roles
have changed and the reading coach is not involved instudent/classroom reading except
to test, and submit reports to monthly.
16 The ARI support just gives up more and more to do, but they have forgotten what it is
like to be in a classroom teaching! They need to pick up these students daily!!!!! We need
more time with our other students. They are no longer in our rooms as much doing one
on one support. Now it is mostly by grade level
22 Dwindled to nothing. 1
36 Our building did not have a reading coach
868 less support over the years
3 7 We do not have a reading coach.
866 A lot. They are K-12 and have fewer staff I miss them
At first, ARI couldn't do enough for us and now I never hear from ARI. Reading coaches
may receive help from ARI but the classroom teachers do not-! would like to have some
fresh ideas to use in the classroom.
858 We have received less support training in the past several years
85 7 It has lessened
855 less support
854 It has progressively gotten worse.
853 I have less support since we adopted Reading Street. It is a good program, but it is
not a ''fix all". I enjoyed the various lessons/strategies we saw demonstrated by ARI reps.
31 I believe in ARI and the strategies and skills that it offers. As I have moved into a role
that is now serving three schools instead of one, I feel that its' effectiveness has been
compromised. The work load of three schools has negatively impacted the elementary
school and the amount of coaching that I am able to do a the elementary level. There is
not enough time for one person to effectively coach multiple schools (not ifthe coaching
is centered around the foundation coaching research of ARI). While, I do see the need for
literacy coaching at the middle and high school level, I feel that the foundation of all
literacy coaching should be centered around the elementary grades. The data at my
elementary school show the negative impact of being pulled out of the elementary
school. This data reflects the gap we are seeing at the middle school which will transfer
to the high school. If we don't close the gap at the elementary level, gaps will continue to
widen.
821 !feel the classroom support is not what it once was. In the beginning, our Reading
Coach would come in the classroom and work with a group of children in one corner
while I was in another. She would pull small groups during the day for more reading
intervention and use a variety ofprograms. We do still have a group that is pulled out for
Reading Intervention, but !just feel that it is not working like it once did. !feel their are
so many time issues and scheduling problems that we get caught up in the minutes of the
day rather than helping the child. If I need 20 more minutes on reading or 10 more
minutes with a group, I should not feel stress or anxiety about being off my schedule. A
good teacher teaches, and I believe that can not also fit a schedule!!
 AR1 EVALUATION: ACE 29

201 !feel as though ARI is viewed negatively among teachers. I am not saying that I
agree with them, !just want to give them a voice. Teachers I talk with and comments I
hear or read seem to resent what they deem as people "out of touch with reality"
constantly telling them how to do their jobs better. I am not simply referring to regional
support staff but the state department personnel as a whole. Also, in my observation, all
schools have a unique culture, strengths and weaknesses. I think it would be a benefit to
have support staff who understand these situations. For example, what a teacher has to
do at a K-12 school where a high school teacher may have to teach 5 different grade
levels or possibly 2 different subjects to 5 different grade levels may have a harder time
planning wonderful, engaging, effective lessons incorporating the lasted innovative
strategies and have meaningful conversations about data and students work with
colleagues during their 47 minute prep time than let's say a teacher at a larger school
who teaches 8'" grade science all day, year after year and may have multiple teachers
who teach the same thing in which they can actually collaborate with someone or at least
share responsibility for lesson planning. All schools are not created equal just as all
students are not the same. I would love for someone to actually come to our school and
show us how to do everything we should be doing in the amount of time we have and with
the resources and students we have in our area. I truly do not mean that in a sarcastic
way .. .I am very serious! I don't need professional development.. we need someone to
literally get engrossed in our culture and help us figure out how to do things in the most
effective way. That would be my dream! I do not look at the state department as
something scary or threatening. ..I truly believe that everyone is doing the best they can
with what they know. I hope these surveys are beneficial and I hope people will truly take
time to express their honest opinions. I talk to too many people who want to see changes,
however, no one whats to be the one to actually do the talking to anyone above
them ... they just wait until their retirement and the cycle repeats itself
64 For the most part, I think the design and emphasis ofARI have continued to develop in
a largely positive manner. However, I think the design was particularly effective several
years ago, when there seemed to be a heavier emphasis on providing regular support to
school based coaches. Over the past 2-3 years it seems that ARI coaches have been
continually asked to do more with less, stretching their resources and reducing their
impact. In order for ARI regional coaches to be most effective they need to A7lOW their
teachers and schools deeply. It takes extensive time to build necessary levels of trust and
knowledge. It doesn't appear that ARI regional coaches have been afforded that time for
the past several years.
74 The changes to ARI have had a negative effect over time. When ARI was more
involved with the districts and schools, reading scores were really moving not only in my
school but our district as well. I have seen the ARI staff shrink and their support has had
to target a smaller population. By analyzing the reading data in Alabama, it is clear that
with a strong ARI presence and leadership, we can improve reading instruction.
However, reading instruction will not improve and be sustainable without a healthy ARI
staff that can be hands on. I know this requires money but what we value we invest. If we
want to increase literacy we need to go back to training and supporting our teachers. In
the "old days, hanging the ARI banner meant something. It meant instructional strategies
 ARI EVALUATION: ACE 30

were aligned to best practices. Support was in place for the principals, coaches, and
teachers. Right now, the support is great but we need more. We need to require training
for ALL teachers. Make it a priority. Make follow up by principals a requirement to
having a paid coach in their building. Right now, the state is giving everyone a coach but
not having any requirements along with it- just suggestions. MAKE principals attend
training. They must know what they are looking for in the classrooms. We CAN conquer
this as a state but go back and look at when reading was improving the most - ARl was
up front and driving! I
0 In some respects I love the changes to ARI! But in others not so much. I think some of
the best teaching I saw was when teachers were going through the rigorous professional
development and certification process ofARIIARA. The best practices learned, applied,
and monitored were so helpful not only as a professional community, but most
importantly on the impact of our students. New teachers and veteran teachers would
benefit from having more support and more ongoing professional development from ARI
on best practices and the implementation of effective reading strategies. I wish there was
a stronger presence of this in our systems and our schools.
22 As far as my reading coach is concerned, at the beginning of my career my reading
coach worked with students. She was able to pulled my tier III students daily for
instruction. I saw huge gains made in each child. Reading coaches have now become
Instructional coaches. They are now responsible for all subject areas. My Instructional
coach now only works with teachers instead of students. I think this is a huge disservice
to students

2. However, reading coaches were more favorable about the change to the K-12
consultancy model, with a majority indicating that the consultant role was valuable:

I love that ARI has grown with the changes to the new standards. I believe that this has
greatly impacted their ability to stay effective in the most changing times in education.
Our regional coach has provided tremendous support to us, as school coaches, to build
our knowledge of best practice and to help us understand the depth of the changes to the
standards. She has led us in a number of book studies to deepen our knowledge and come
into our schools to help us put new ideas into practice. We have in turn been able to pass
that knowledge base down to the school level.
63 There have been changes in the focus from reading to best instructional practices
across the curriculum (math, social studies, and science) but I believe that this has been
a good change. However, I do miss the opportunity to spend time with small groups of
children that I provide intervention to.
111 IT'S MORE EFFECTIVE TODAY BECAUSE THERE IS MORE OF A FOCUS ON
IT AND MORE PD WITH IT. IT'S NOT JUST THE READING COACHES ANYMORE...
THE CURRICLUM SPECIALIST NOW GETS TRAINED AND IMPLEMENTS IT
WHICH HELPS BUILD CAPACITY WITHIN THE SCHOOL. I DID LIKE IN THE PAST
WHERE MEETINGS ALSO INCLUDED A LEAD TEACHER SO THERE WAS A REAL
ARI TEAM AT EVERY SCHOOL
 ARI EVALUATION: ACE 31

3. One unintended consequence of the move to consultant K-12 coaches appears

to have been that a number of teachers without specific expertise in reading
were labeled not only reading experts but de-facto content experts as well,
which made them much less effective. Coaches who responded to the survey had
in the majority less than five years experience as a reading coach.

239 As the years have passed, much of the training seems to have been forgotten or
neglected. First grade students at my school have "comprehension problems".
However, the root of the problem is that these students are not accurate or
automatic. There is a p ush in depth of knowledge and discourse while the focus
should be on creatingjluent readers.
When students struggle with every other word, there can be no comprehension. I
fee l that our primary school (K, 1st &2nd grade) has lost its focus. We hear that
only a third (approximation) of the 3rd grade students met the goal in the current
3rd grade state assessment. I question whether this is a comprehension issue in
regard to not understanding the text or whether these students are not fluent readers
and cannot just read the test or answer choices. Half of our current faculty has not
had training at a school/site level. I am sad that a program with so much positive
impact and research seems to be given little thought in my XXX County school.
185 In the beginning before Dr. Bice, the focus was Direct Instruction for all
readers with complete fide lity to the scripted reading program =(Now the focus has
shifted to differentiated strategies for all students; readers and non-readers =).
203 In my opinion, I am more effective as a reading coach than a math coach. I
have received extensive and in-depth training/professional development as a
reading coach. Recently, my title has changed to instructional coach which includes
math in addition to reading. I have received very little training/professional
development as an instructional coach in regard to math. Therefore, in my opinion,
I am unable to be an effective math coach.
328 It isn't effective at all. The original ARI strategies are old and much more
support is needed to support 21st century literacy skills. The biggest problem is the
coaches only know reading for the most part and can't support literacy instruction
in other core content areas I I I I I I I They don't know how to be instructional coaches
95 We do not need reading coaches. They should be tutoring low level students.
Instead, our reading coach has a room to herself and does nothing but a few data
meetings every once in a while. Also, they are not in the classroom teaching and do
not have "real classroom" experiences. They give veteran teachers things to do that
veteran teachers know is a waste of learning time. I have taught for many years. I
am speaking from experience. We do not need reading coaches. New teachers need
mentors, which we already give them (veteran teachers). Reading coaches are a
waste of money and resources. They should be in the classrooms teaching. Teachers
do not need them. Teachers can just attend training on ARI when they haven't had it
and go from there. Reading Coaches just have a "cushy" job like a hall
administrator and do not make any impact on student learning. They push some
paperwork for the Principals and that is it! It is time to put the Reading Coaches
 ARI EVALUATION: ACE 32

back into the classrooms as a regular teacher, although they would hate that since
they have had such an easy job as Reading Coach. Our reading coach has less years
teaching than most of the other teachers in the school where I teach. She knows very
little about how to handle a classroom. I have seen this in other schools also. Put
them back in a classroom and let them make a difference with teaching. We not
longer need them!!! We do not need them for any testing or classroom monitoring
either because the classroom teachers already has to do all of it.
73 Our reading coach has been out of the classroom too long and is out of touch
with reality. She shoves data and "solutions" down our throats without any concern
for the practicality or even chance of success involved. She is very vocal about what
we should do, but is not one to "get her hands dirty". She is a waste of a teacher's
salary.

4. Some teachers objected to ARI methodology and practice, which they saw as
in disagreement with best practice they had been exposed to either at the
university or in other forms. In particular, the changing focus from the original
communities of practice, K-3 building-based focus to a basal-reader system
demanding a strict fidelity and discounted the teaching of writing was discussed
as negative. Some teacher indicated that the emphasis on a "fidelity to basal"
system took away their ability to respond to individual students and their
enthusiasm for ARI:

139 When I first became a reading coach, I served one school. I felt I was quite
effective with teachers in this setting. Then a couple ofyears ago, my district
decided I would become an "instructional" coach and coach both reading and math,
even though I had not taught math in over ten years. I did not receive any training to
help with this deficit. Then this school year, my district decided to make my position
a "zone instructional coach". I am now responsible for grades K- 12 in all subject
areas. My district has chosen to not use AR1 and our students have suffered because
of this decision229 Every time a regional AR1 coordinator changed, the expectations
were changed. We went from progress monitoring all of the time to "teaching to
fidelity". Then we were told that writing is not a part of reading instruction, 3 years
later wewere told to add writing back to the reading program/centers. Teachers
need to be given more time to plan and implement instruction that they know works
for their individual classrooms/students. Teachers are professionals, our opinions
and knowledge should be an important factor in how to instruct our students.
256 After our school went through AR1 training, teachers and students thrived. The
following year, our county adopted "Teach To Fidelity" along with Scott Foresman.
Everything we had learned in AR1 went out the door. It was such a discouraging
time to be teaching. The support was only about data.
347 I've been through 4 coaches. Much of this "support" (with the exception of our
present coach) was nothing more than changing everything up (including the
jargon) in midstream and confusing teachers, students, and parents (and sometimes,
 ARI EVALUATION: ACE 33

it seemed, the coaches). What began as an exciting opportunity to better reading

turned into a nightmare of micromanaging theft of methods that made sense to
students, parents, and teachers and theft of EVERY SINGLE BIT of respect for the
training and experience teachers received in college and through years ofpractice.
These were replaced with intrusions that did not allow for ANY classroom
autonomy. Also askew was the removal - expressed removal by coaches, district
supervisors from ARI, and principals alike - of the teachings of Social Studies and
the Sciences in the early grades under the guise of "But Reading will help them
improve on these later on ". What was outrageous to me was that after following
ARI's methods, I saw no improvement with struggling readers after ARI than the
struggling readers I taught before ARI using the researched based methods learned
in college and post-graduate studies (in fact, at times, I saw regression from what I
believe would have been progression had I been allowed to use different methods) ..
The current support is that - SUPPORT, and it is greatly appreciated! I cannot help
but wonder, however, if this will last or if we are just in the proverbial eye of the
storm; I do not know, because, the bottom line is this: I do not trust ARI with the
education of our community's youth.578 Initially, it was extremely intimidating and
judgmental when reviewing student scores. It slowly transitioned into more of a
review I troubleshooting time to reflect and tweek teaching techniques. I think we
still WAY OVER TEST THESE STUDENTS! We need to narrow down the testing to
provide more teaching time. It's overwhelming, an inefficient use of time, and
ridiculous to test the same skills with several different sets of tests. These poor kids
are exhausted and not enough time is provided in the schedule for actual teaching. I
don't know one teacher who is isn'tfrustrated by this.
437 ARI started out very helpful. However, it is nothing like it started out being.
Teachers are not able to teach!! Th ey are expected to do more with less. On top of it
all, there is now another body (reading coach) in the room that can't even help with
struggling students because it is not in their job description!
850 Th e thing that bothers me and other faculty members the most is that the
reading coaches do not work with children. We do not have any interventionists at
our school. Therefore, the children are the ones who are being hurt becausethey are
not getting the extra help that they need to move to the next level as a reader. We
need more interventionists at al schools across the state instead of reading coaches!
54 the support hasn't changed, only the asinine things they've told us to do has
changed: concept boards(JOKE), the "don't have the students raise their hands to
answer" strategy(REALLY?????), the "don't ever vere away from the script in the
book" nonsense! Let great teachers teach!!!!! Let good teachers receive mentoring
from great teachers!!!!! Fire the bad ones!!!!!! Come on Alabama .... . It's not that
hard!!!!!!!
 ARI EVALUATION: ACE 34

Black Belt Case Study

In data presentation it is often useful to illustrate a particular effect of an initiative on a

well-known geographic or cultural region within the state. For example, in New York
State, results of program impacts sometime examine differences in Upstate and New
York City scores. In Michigan, the Upper Peninsula represents a unique cultural,
demographic, and economic region that is identifiable from and atypical to much of the
rest of the state. In the current evaluation, an obvious choice for such a data presentation
is the Black Belt counties of Alabama. As Winemiller says in the Ency clopedia of
Alabama:

Alabama's modem Black Belt region, and the Southern Black Belt in general,
continue to be defined by the legacy of slavery and the plantation agriculture
unemployment, low-achieving schools, and high rates of out-migration. The
Black Belt has a high number of single-parent households, high teen birth rates,
and poor Democratic Party. Numerous similarities can be drawn between the
Black Belt region and developing nations in both Latin America and sub-Saharan
Africa, not the least being extractive economies, plantation systems, and heavy
dependence on more developed economies. Traits usually associated with
agrarian society such as large family and household size persist within the region
today. Winemiller, 2009, n.p.

An additional dataset was generated using just the 18 county and six city schools
considered to be in the Black Belt region. These school systems with their school codes
are: Barbour (003), Eufaula (133), Bullock (006), Butler (007), Choctaw (012), Crenshaw
(021), Dallas (024), Selma (191), Greene (032), Hale (033), Lowndes (043), Macon
(044), Marengo (046), Demopolis (128), Linden (168), Montgomery (051), Perry (053),
Pickens (054), Pike (055), Troy (199), Russell (057), Phenix City (184), Sumter (060),
and Wilcox (066). Historically, these counties, cities, and LEAs have experienced greater
poverty than the average equivalent in Alabama along with a host of other socio-political
issues beyond the scope of these analyses. It is an understatement to say that these are
underserved populations within the state. It is, therefore, of little surprise to note that
these LEAs have some of the lowest literacy rates in the state.
 ARI EVALUATION: ACE 35

Figure 4. The Black Belt region of Alabama (from Wikipedia) .

Schools within the Black Belt region were examined separately after conclusion of the
total state analyses to test for any differences. The same measures were implemented
regarding overall literacy changes and per-pupil expenditure influence. The one-way
ANOVA also found a significant, positive relationship between year and literacy (F(s) =

9906.60,p < .001) with significant, positive differences found between each year. Of
note, ARMT scores were also positively correlated with per-pupil expenditures (r = .71 , p
= .02). However, further analysis demonstrated that this was collinearity between per-
pupil expenditures and year for these school districts . When accounting for time, costs
were no longer significant as an explaining factor.

Upon full implementation of ARI, these LEAs produced most of the lowest literacy rates
in the state, a full nine percentage points below the state average. However, these schools
 ARIEVALUATION: ACE 36

not only progressed along the same linearity as the rest of the state, but did so at nearly
150% the rate of improvement for the rest of the state. It is further noteworthy that the
majority of these gains occurred in the last few years of the study, which argues that as
more students received ARI instruction, more benefit to this underserved population
became apparent.

Figure 5. Alabama literacy rate (ARMT) between Alabama and the Black Belt Region.

Annual Literacy Rate (ARMT)

between Alabama and the
Black Belt Region
90%

80%

70% :::: ! ! ~
60%

50%
2006 2007 2008 2009 2010 2011 2012 2013
-+- State - Black Belt
 ARl EVALUATION: ACE 37

A Final Note before the Summary:

This report would be incomplete without including the document below, provided to the
Alabama Legislature based on recommendations gleaned from the same data on which
the current evaluation is based:

The Alabama Reading Initiative, 2016-2017

The Alabama State Department of Education (ALSDE), in response to an analysis of
student performance data, has revamped its Alabama Reading Initiative (ARI) and shifted
its focus back to foundational reading and literacy skills in Grades K-3. It was determined
that schools falling below the state average in 3rd grade reading would be identified as
Tier II schools and will receive full funding for a reading specialist. The ARI regional
support staff will work side-by-side with the school reading specialists in order to
improve student performance in the identified Tier II schools.
The following has been implemented by the ALSDE and ARI with local education
agencies (LEAs) for the 2016-2017 school year:
0 Defining ALSDEIARI Support- The ARI will fully support schools falling below the
state average in 3rd grade reading and identified as Tier II schools.
0 Restructuring ARI regional support staff- ARl regional support staff was restructured
resulting in a reduction ofstaff (67 in 2015-2016 to 27 in 2016-
2017); regional support staff work was redefined to serve only Tier II schools.
0 Funding for Tier II School Support - ARI allocations to LEAs will fully fund a reading
specialist in residence at each designated Tier II school.
0 Supporting Tier II schools - ARI regional staff will be assigned to fully support the
school-level reading specialist in each Tier II school; a new job description specific to
the Tier II school reading specialist was developed to include minimum skills applicant
must possess.
0 Instituting an LEA/School Memorandum ofAgreement (MOA) - Tier II schools must
submit to the ALSDE an agreement (MOA) that outlines annual improvement goals in
3rd grade reading and required participation in professional development.
0 Collaborating to design ARI Professional Development (PD) Plans -ARI PD plans
are now a requirement for Tier II schools; these will be developed collaboratively with
the LEA and ARI regional support staff then reviewed by ALSDEIARI.

(Note: It would, of course, be useful to capture the impact of these changes by collecting data in the form of
a program evaluation focused on the changed aspects of the program to examine whether they had
enhanced delivery of support to teachers and students, and if there is a longitudinal impact on student
achievement scores.)
 ARIEVALUATION: ACE 38

Summary of Findings:

This evaluation began by examining the historical goals of the Alabama Reading
Initiative and by analyzing data to determine how well the ARI has met its original goals,
what changes have take place over the last 18 years of the program, and how effective the
program has been in establishing meaningful support for literacy in the state.

Outcomes
1. Commit to a 100% literacy rate among students.
Support/Sustainability Factors
2. Commit at least 85% of the faculty and administration to attend a two-week
intensive summer institute focused on reading improvement and ongoing, year-
round professional development.
3. Employ at least one full-time reading coach, whose job was to work with teachers
as well as with struggling readers.
4. Collaborate between schools and higher education faculty partners, who served as
mentors, provide access to research and help solve instructional problems related to
literacy and learning.
5. Engage in partnerships with local businesses.
6. Protect the reading block in the master schedule

Data do not exist within the scope of this evaluation to examine the extent that business
partnerships and master schedule protection of reading are ongoing aspects of the
program, (although survey data suggest that the reading is no longer protected in the
master in at least some schools), inferential statistical data analysis reveals the following
about the literacy rate outcome:

1. Although the data show that reading achievement scores have improved for students
across the state, especially in the poorest schools, the primary goal of achieving 100%
literacy has remained unrnet. However, ARMT scores ranged from a low of 79% in 2006
to a high of86% in 2013.
 ARI EVALUATION: ACE 39

2. Schools in high relative poverty areas (LEAs with greater than average rates of
free/reduced lunch) had students with significantly lower proficiency scores (r = -.11 , p <
.001).

3. Students from low-achieving schools are receiving the most benefit from ARI. There is
an inverse correlation between literacy gains and initial literacy rates (r = -.24, p < .001).
Students in the Black Belt of Alabama were more likely to make gains than students in
other areas of the state.

4. Though data indicate a positive impact of ARJ on student achievement, data also
indicate that no one factor (staffing of ARI, funding levels of the program, or curricular
changes) appears to have had a singular or explanatory impact on student achievement
scores.

Data analyzed from the survey results yielded these findings :

5. Nearly every respondent who addressed the question said that ARI was a successful
and valuable program overall and that it had a positive impact on teacher knowledge and
development, the spread ofbest practice and in the K-12 system, and as an initiation to
new teachers in the district to a common language about teaching reading.

7. Reading coaches were much more favorable about the change to the K-12 Consultancy
model, with a majority indicating that the consultant role was valuable.

8. One unintended consequence of the move from K-3 to K-12 coaches appears to have
been that a number of teachers without specific expertise in reading became not only
reading coaches but de-facto content experts as well, which made them much less
effective. Coaches who responded to the survey had in the majority less than five years of
experience as a reading coach.

9. Some teachers objected to ARI methodology and practice. In particular, the changing
focus from the original communities of practice, K-3 building-based focus to a basal-
reader system demanding a strict fidelity had many negative comments. Some teachers
 ARI EV ALUATIO : ACE 40

indicated that the emphasis on a "fidelity to basal" system took away their ability to
respond to individual students and their enthusiasm for ARI.

10. The original factors thought to be important to the success of ARI and included in the
original supports such as the role of higher education consulting either received virtually
no mention in the data or had been changed as the program became actualized. Teachers
no longer have the opportunity for two-week trainings, and many survey respondents said
that was an important factor in whether ARI was a central part oftheir school 's
curriculum. Finally as the data have shown, changes in staffing meant the original goal of
placing reading coaches focused on improving reading in K-3 classrooms has changed
several times, as have the foci and pedagogical approaches.

Recommendation:
In addition to recommending an ongoing purposeful external evaluations of such a large
state program, the major findings from this report suggest an examination of the entirety
of ARI, perhaps by a blue ribbon panel of experts in the field of reading. It should be
evaluated in regard to its fidelity to the original mission of the program, to issues ofhow
professional development is delivered, the extent to which teachers and children in
Alabama schools are being supported, and the extent to which the attendant costs of the
program are in line with its benefits to the children of Alabama.
 ARl EVALUATION: ACE 41

References

Black Belt of(region of Alabama). (n.d.). In WiA:ipedia. Retrieved January 10, 2017.
https://en.wikipedia.org/wiki/Black _ Belt_(region_of_ Alabama).
Dick, R. (n.d.) Grounded theory: A thumbnail sketch. Retrieved September 9, 2002 from
http://www.scu.edu.aulschools/gcrn/ar/arp/grounded.html
Glaser, B. & Strauss, A. (1967). The discovery of grounded theory. Chicago: Aldine.
Institute ofEducation Services. National Assessment of Education Progress (NAEP).
Retrieved from http://nces.ed.gov/nationsreportcard/glossary.aspx#percentile
Moscovitch, E. (2001). Evaluation ofthe Alabama reading initiative (Final Report) .
Retrieved January 10, 2017 from http://files.eric .ed.gov/fulltext/ED464366.pdf
Oranika, P , (20 16). The Alabama reading initiative program evaluation. Unpublished
report. Alabama State Department of Education: Montgomery.
Pandit, N. (1996). The creation oftheory: A recent application ofthe grounded theory
method. The qualitative report (2), 4, 1-14.
Winemiller, T. (2009). Black belt region of Alabama. Retrieved January 10, 2017 from
http://www.encyclopediaofalabama.org/article!h-2458