Running Head: EVALUATION DESIGN 1

Evaluation Design

Jordan Cummings

College of Liberal Arts, Auburn University

POLI 7520: Program Evaluation

Dr. John C. Morris

November 27, 2023

Running Head: EVALUATION DESIGN 2

Program Description

To further improve Alabama’s educational accountability systems, in 2017, the state

adopted a consolidated State plan following the Every Student Succeeds Act (ESSA) (State of

Alabama, 2017, p. 3). The ESSA replaced the Elementary and Secondary Education Act, and it

serves the purpose of ensuring all students in public schools are provided with a more equitable

and higher-quality education (Lee, 2018, p. 2). As a part of this plan, one area of focus for school

districts is increasing the percentage of students who graduate from high school. This essay will

put forth a plan to assess whether or not, based on the current trend, the percentage of

non-graduating students will decrease by 50% by the year 2030. How this is assessed is that the

state determined a baseline graduation rate (in this case, Alabama used the 2015-16 graduation

rates), determined the percent of non-graduates by subtracting the percent of graduating students

from 100, and then divided that number by two and added it to the baseline statistic to get their

goal (State of Alabama, 2017, p. 17). For example, the baseline percent for “All students” during

the 2015-16 school year was 87.12% and to decrease the number of nongraduates by 50% by

2030 the percentage of graduating students should be around 93.62% (State of Alabama, 2017, p.

18).

Description of Policy Theory

Some of the practices that this act forces schools to take part in to address graduation

rates are: tracking early-warning indicators in Freshmen (such as having enough credits to

advance to Sophomore year), focusing on attendance data, creating a culture of shared

responsibility for all adults involved with the students, raise standards for student achievement,

easing the transition from middle school to high school, and assessing and addressing

disciplinary practices that historically have had a higher impact on disadvantaged students (State
Running Head: EVALUATION DESIGN 3

of Alabama, 2017, 34). The implementation of this act put forth the idea that the aforementioned

indicators are of great importance and impact the graduation gap. For obvious reasons, students

failing classes in their Freshman year or having an excessive amount of absences is cause for

great concern. However, the act also takes into account socio-political issues such as the issue of

students from diverse backgrounds receiving more harsh and strict punishment than their

counterparts. Actions such as these are shown to have a great impact on a student's success and

the act requires schools to recognize and address issues such as these. The ESSA also recognizes

that there needs to be more highly qualified educators working in what one might call

underprivileged communities. To address this issue, Alabama allocated a significant amount of

money to incentivize teachers to become National Board Certified and then work at schools that

are in great need of high-level teachers (failing schools, schools with low graduation rates,

schools that experience high levels of poverty) (State of Alabama, 2017, 41). With all of these

measures in place and more, Alabama’s plan to increase graduation rates takes a

covering-all-angles type of approach.

Information Gathered/Stakeholders

At the end of each academic year, data regarding the number of students who proceeded

on to the next grade, on track to graduate with their starting cohort (the percent of students on

track to graduate on time); students who failed a core course; students who have not met a

college or career readiness standard by Junior year; the number of certified and nationally

certified teachers; and, attendance and discipline data – particularly for students at risk of not

graduating – will be collected from the school administration and sent to the state department

where all evaluation data will be held. This data will be gathered to assess whether or not the

schools are keeping up with the planned trajectory and if the indicators the state’s plan addresses
Running Head: EVALUATION DESIGN 4

are truly impacting the student's achievement levels or not. Stakeholder surveys will also be

collected to gain qualitative insight into the effectiveness of the state’s plan. Stakeholders to be

addressed can be divided into two roles (many of which are shared): beneficiaries and victims.

The beneficiaries of Alabama’s plan are parents, teachers and other school staff, schools

and school districts, the Alabama Department of Education, the U.S. Department of Education,

colleges, the workforce, and many more. Most of these bodies similarly benefit from this act – an

increase in high school graduates prepared to take on the next step in life (whether that be

college or a career). However, many of these beneficiaries could also be seen as potential

victims. For example, schools and school districts have to spend an exorbitant amount of time

and effort collecting data. This stress to make sure they comply with federal regulations is also

added to the state department. In a way, some students might be seen as victims, too. For

example, if National Board Certified teachers are incentivized to go to “failing” schools, then

schools not labeled as “failing” might be losing out on the opportunity to have a highly qualified

educator.

Analysis Plan

In order to evaluate the effectiveness of the ESSA’s goal of increasing graduation rates to

90%, the best model to use would be a pretest/posttest style evaluation. This evaluation will take

on a quasi-experimental design, as it will start with a pretest of Freshman from twelve different

schools in six different regions of the state (one Title I school [that is required to enforce the

state’s plan] and one non-Title I school [that is not required to enforce the state’s plan] from each

region [NE, NW, E, W, SE, and SW]) for the 2023-24 school year. Every year after, the data will

be collected on the students and compared to the state’s target in a post-test. This will be

followed through for four years to guarantee the effectiveness of the program.
Running Head: EVALUATION DESIGN 5

A quasi-experimental method would best fit this evaluation due to the nature of

fluctuation in the field of education. With a school that is not required to enforce Title I, the

achievement of schools who are required to enact Title I will have a basis to be compared to. It

can be assumed that if a school is already highly achieving then it will maintain such a state. If

there is any sudden change in their achievement, then the data can be analyzed to see what

changed during that year. Compared the non-Title I school, whose graduation rate should be

relatively steady, the data collected during the 2025-26 school year for the Title I school should

reflect a graduation rate of at least 91.62% for the state to be on track with its goal (State of

Alabama, 2017, p. 19). If the non-Title I school stays steady and the Title I school increases then

the null hypothesis can be rejected. The reason for this is that the non-Title I school serves as a

sort of control. If they do not change, we can assume there have been no external conditions that

might impact graduation rates across the state. Therefore, if the Title I school does change in this

case then we can assume that it is the state’s plan that has made this impact.

Timeline

For a detailed Gantt chart displaying the timeline of the evaluation, see Table 1 below.

For this evaluation, data collection will start with the graduating class of 2022-23. From that

class, we will gather baseline data to see if the plan is on track for its 2030 goal. After that, for

the next three years, we will gather data regarding attendance and disciplinary records, the

survey answers from the stakeholders, as well as the yearly graduation rate and compare it to the

state’s overall goal to ensure the effectiveness of the program. The state Department of Education

will receive yearly updates during December, and in December of 2026, a final report will be

made determining the effectiveness of the state’s plan and a determination of whether or not they

will meet their 2023 goal.

Running Head: EVALUATION DESIGN 6

Table 1

Conclusion

To evaluate whether or not the state of Alabama will reach its goal of decreasing the

percentage of non-graduates in the state by 50% by 2030, the graduating class of 2026,

throughout the state, will have extensive data collected regarding the areas of effect from the

state’s plan. Schools implementing the plan will be compared against those who are not to ensure

that the effect seen can be attributed to changes enforced by the ESSA. The data collection, itself,

will be heavily placed on the school districts whereas the evaluator’s job will be to analyze all of

the collected data to see if the trends match what the state has projected.

The main threat to the validity of this evaluation is a misrepresentation of data reported

by the school districts. However, paired with the qualitative surveys from stakeholders, this issue

should be easily recognizable in which case the evaluator can do an audit of the data to check

validity. The questions for the stakeholder survey serve two functions. The first is to see if

students are just being pushed through the system – to see if they are being prepared for
Running Head: EVALUATION DESIGN 7

graduation or not. If not, then the plan is failing its goal even if its percentages are increasing.

The second is to provide a check for the districts. If a district is reporting extremely well but the

qualitative data comes back showing most of the students report subpar education and

preparedness then the evaluator will know that there is a discrepancy that needs to be addressed.

References
Running Head: EVALUATION DESIGN 8

Lee, A. M. I. (2018, June 20). Every student succeeds act (ESSA): What you need to know. The

Coordinating Center.

https://www.coordinatingcenter.org/files/2018/09/Every-Student-Succeeds-Act-ESSA-W

hat-You-Need-to-Know.pdf

State of Alabama. (2017, October 12). Revised state template for the consolidated state plan:

The elementary and secondary education act of 1965, as amended by the every students

succeeds act. Office of Elementary and Secondary Education.

https://oese.ed.gov/files/2022/01/AL-ESSA-State-Plan_Redacted.pdf