Program Evaluation

The Accelerated Reader Program at ABC Elementary School in Statesboro, Georgia

Chris Kahley Myra Lipsey Danielle Walden

FRIT 8435 Fall 2012

Georgia Southern University November 30, 2012

Table of Contents
I. Executive Summary3 a. Recommendations.4 II. Introduction.5 a. Limitations6 III. IV. V. VI. VII. VIII. IX. Focus of the Evaluation8 Overview of the Evaluation Plan and Procedures.9 Presentation of Evaluation Results.10 Stakeholder Perceptions.11 Conclusion and Recommendations.13 References15 Appendix16

Executive Summary
The purpose/objective of our evaluation was to determine whether or not the Accelerated Reader program featured at ABC Elementary School in Statesboro, Georgia, shows gains in reading scores using the STAR reading test. Other correlations are also being considered such as CRCT test scores and MAP test scores. The Accelerated Reader Program is used to motivate students to read a variety of books. After students have finished a book, they take a test to see how well they comprehended the material they read. By using both qualitative and quantitative data to analyze the effects of this program, this study shows that the Accelerated Reading program is beneficial to students and their reading scores both on the STAR reading test, as well as the MAP test. There does not seem to be a direct correlation of performance on Accelerated Reading and the CRCT test scores however. The following evaluation questions were created in order to serve as a guide to provide this information: Do the Accelerated Reader participants demonstrate learning gains in reading over the study period? Are variations in student participation levels in Accelerated Reader associated with differential learning gains?

The Renaissance Learning database was accessed in order to collect data for the reading comprehension and reading level of all participating fourth grade students for the school year 2011 2012. A survey was distributed to all fourth grade teachers as well as school administrators and the media specialist, in order to collect stakeholder opinions about this program. Evaluation findings show that the Accelerated Reader Program at ABC Elementary School:

Provides an opportunity for students that use this resource to improve their STAR reading test scores, thus which improves MAP test scores from the beginning of the school year to the end of the school year. Provides a valuable way for teachers to differentiate their teaching style to make learning more individualized for students on a variety of reading levels. Provides a fun and self-monitoring way to motivate students to not only read, but comprehend a variety of books.

Recommendations
Although there are many factors that lead to limited evidence, the data does disclose evident considerations about the Accelerated Reader Program at ABC Elementary School: The teacher surveys show an encouraging reaction to the program and that some teachers view the program as beneficial to student reading comprehension and participation. Student records indicate that there is a gain in STAR test scores, when the Accelerated Reader program is implemented by teachers and students use it appropriately. Student records also indicated that there is a correlation between Accelerated Reader, STAR test growth scores, and MAP test scores, but not necessarily so with CRCT test scores.

The data presented in the evaluation report shows the benefits of the program. As a result, the evaluators make the following recommendations: It is recommended that the Accelerated Reader program be continued so that students are given the opportunity to use a resource in which they greatly benefit. It is recommended that teachers meet with the media specialist to learn how to better implement this program in order to achieve greater results. It is recommended that students be given ample opportunity to use this program during downtimes in instruction in order to provide them the chance to increase their reading comprehension.

It is recommended that all students be provided with a variety of motivating factors to either start using or continue using this valuable resource.

Introduction
The purpose of this evaluation was to determine if the Accelerated Reader Program (a part of Renaissance Learning) is helping to raise student STAR reading test scores (which in turn will show an increase in comprehension as indicated on the MAP test as well) among fourth grade students at ABC Elementary School. At the fourth grade level, students are becoming more independent readers than they were in earlier years. Coursework is at such a rigorous pace, students in fourth grade do not have as much time to devote to accelerated reader during the school day as they did in earlier grades. Therefore, students are required to complete most of this reading at home. It is because of this change in the way accelerated reader is used, that we felt using fourth grade would be advantageous to this evaluation. We looked at growth reports to compare student scores from the beginning of the school year (August 2011) to the end of the school year (May 2012). We also looked at diagnostic reports of students reading practice quizzes to help make a determination if students are increasing their comprehension. This in turn would show an increase in MAP test scores at the end of the school year. Teachers surveyed, felt there was a correlation between Accelerated Reader and STAR and MAP test scores. They did not feel there was such a close fit between Accelerated Reader and CRCT test scores because CRCT is not individualized as is Accelerated Reader and STAR and MAP. The audience for this evaluation is the faculty, administrators, parents, and students at ABC Elementary School. The school serves students in pre-kindergarten to fifth grade. The total population is approximately 550 students. We chose to use fourth grade only to get a random

sampling of the student population. The school is located in Southeast, Georgia, in a small, rural town. The school is a Title I school made up of 80 % free and reduced lunch. The faculty is composed of teachers with degrees from Bachelors to Specialist. Of the 31 teachers, at least 20 have a Masters degree. Each pre-kindergarten and kindergarten classroom has a paraprofessional. All of the paraprofessionals have a minimum of a high school diploma.

Limitations
It would have been impossible to focus our study on the whole student population of ABC Elementary School. There would not be enough time to look at STAR test scores and growth in reading comprehension for all 550 students; therefore, we limited our evaluation to fourth grade only, which is approximately 85 students. It was difficult getting the teachers to return their surveys. Reminder emails and notifications were sent to teachers, but with so many commitments, teachers forgot about the surveys and had to be reminded several times. We did finally receive all surveys. Due to time constraints, surveys were not distributed to parents of the students or the students themselves. In retrospect, sending letters home would have been the best way to have obtained feedback from the parents. Student surveys were not conducted either. Creating student surveys and having teachers distribute them to their class would have been a good approach to have used. We see this as a flaw in our evaluation. We needed input from parents and students because teachers are sometimes biased in their feelings towards the Accelerated Reader Program. Surveying parents and students would have given us a valuable insight from a different perspective.

We utilized the reports from the Accelerated Reader website to help us determine which reports we needed access to. The school media specialist was able to pull the reports for the fourth grade students we were evaluating. Because of obtaining this information in this way, there was no need for us to contact the classroom teachers for information. In the diagnostic reports, we found the students zone of proximal development (ZPD) that each student was reading in. We compared where the students were in late August (2011) to where they were in May (2012) to look at growth on the STAR Reading test which gave us evidence to support the continued use of the Accelerated Reader Program. We also collected MAP and CRCT reading scores from the schools testing coordinator. Refer to the entire report for additional information and to seek clarification.

Focus of the Evaluation

The evaluation object is the Accelerated Reader program. The program is network accessed only at school by students and stakeholders. The program serves to raise students comprehension levels as they work their way through the various reading levels. In the fall, students start out taking the STAR reading test to gain a baseline score of their reading level and comprehension ability. At that point, the students reading teacher sets their zone of proximal reading development (ZPD) and sets a goal for the students to work towards by the end of the nine week grading period. The students are given a range of levels to select books from. This is done so that there will be a good selection of books to choose from when they go visit the media center. Accelerated Reader books are labeled with color-coded dots to help students determine which books they should read. Books range from level 0.1 to 12.9. A ZPD for the average fourth grader might range from 3.6 to 5.0. Students never lose the lower end of their zone, they only add to the upper level of their zone. The zone expands as students read and pass tests with a score of 80 percent or better. Teachers track their students progress regularly. They make sure students are taking at least one or two quizzes per week (depending on their assigned ZPD). ABC Elementary School does not use Accelerated Reader scores for reading grades.

Overview of Evaluation Plan and Procedures

In order to evaluate the Accelerated Reader program at ABC Elementary School, a collection of fourth grade student scores on AR Tests and reading levels was assessed through the Renaissance Learning Database. MAP and CRCT reading scores were obtained from the testing coordinator at the school. Stakeholder perceptions and opinions were considered in conjunction with these scores. These perceptions and opinions were assessed through written surveys. Given the short time span to answer the surveys, there was still a high turnout in responses to the questions provided. The response time to the survey is a good sign of how well this program is received at ABC Elementary School. Qualitative data was collected using a survey printed and placed in staff members mailboxes. This survey consisted of a page of demographic questions about the participating teachers, as well as inquiring about their knowledge of the program, and how they may use it to modify instruction. The responses to the survey were categorized by positive/negative outlooks associated with the program. Quantitative data was collected using a case study design by accessing the Renaissance Learning Database and analyzing fourth grade student reading comprehension scores and reading levels between August 2011 and May 2012. All teaching staff of ABC Elementary was sent a survey to detail the perceptions of stakeholders in regards to the Accelerated Reader program (Appendix A). All AR data of the 85 fourth grade students at ABC Elementary School were analyzed to provide information in conjunction with the previously mentioned reading comprehension scores and reading levels. Jointly, quantitative and qualitative data gathered from the three fourth grade teachers (the sampling that was used for our evaluation) linked to the AR program was analyzed in order to complete this evaluation.

Presentation of Evaluation Results

CRCT reading scores and MAP reading scores were collected from the testing coordinator at the school. STAR reading scores, along with total AR points for the school year 2011-2012 were collected from the Renaissance Learning program for each fourth grade class at ABC Elementary School. The CRCT scores were arranged in descending order and then the students total AR points were listed as well. We then looked at the scores to see if there was a relationship between the higher CRCT scores and the AR points that students received during the school year.
ABC Elementary Fourth Grade Classrooms AR Points (School Year 2011-2012) and CRCT Scores (Spring 2012)

Reading CRCT Score (Fall 2011) 855 850 840 835 828 825 824 821 818 808 800 794 794 794 789 778 767

Total AR Points FY 11-12 13.1 40.1 23.5 13.9 27.7 8.1 15.1 0 12.0 0 16.0 16.9 2.8 8.3 0 11.7 12

After reviewing the CRCT scores and the total AR points listed in this table, there did not appear to be any relationship between the number of AR points and the score on the reading portion of the CRCT. Therefore, points are not a valid indicator of performance on CRCT.

However, in comparing the STAR reading test scores and the MAP test scores, there did appear to be a direct correlation between the scores on both. As previously stated, the survey results indicated that teachers felt AR, STAR, and MAP were more closely knit because they were more individualized than the CRCT.

Fourth Grade Teacher Teacher 1 Teacher 2 Teacher 3

STAR Test Results

(number of students showing growth from Fall 2011 Spring 2012)

MAP Test Results

(number of students whose test scores increased from Fall 2011 Spring 2012)

23 out of 26 students showed growth 20 out of 27 students showed growth 23 out of 25 students showed growth

20 out of 26 students met or exceeded their goal score 20 out of 27 students met or exceeded their goal score 24 out of 25 students met or exceeded their goal score

The chart above clearly shows a direct correlation between Accelerated Reader, STAR Test (growth in reading comprehension and level), and MAP reading test scores.

Stakeholder Perceptions
Given such an adequate turnout in our survey, the evaluators were able to gain great insights into how the Accelerated Reader program provides a wonderful foundation in achieving a higher level of reading comprehension scores. There were no major issues from stakeholders associated with use of the AR program or how well students perform on reading comprehension tests provided by this program. Overall, the staff at ABC Elementary has a positive outlook in regards to using Accelerated Reader. As mentioned in the Executive Summary, the Accelerated Reader program at ABC Elementary has proved to be an asset to student reading comprehension scores.

Conclusions and Recommendations

The evaluative data in the previous section of this report displays student gains in stakeholder perceptions of the Accelerated Reader program at ABC Elementary School. This study makes the following conclusions: Program Strengths:

Accelerated Reader provides an opportunity for students to improve reading abilities and comprehension.
Accelerated Reader is an excellent differentiation technique for teachers to use in order to focus on individualizing instruction and in addressing the reading comprehension needs of students.

Program Weaknesses:
Not all teachers monitor student quizzes as frequently as they need to in order to adjust reading levels. There is a margin of error evident in the program due to some students not being good testtakers. Reading comprehension has been measured by quizzes. If a student is one of those anxious test-takers, their scores may be erroneous.

Recommendations:

It is recommended that the Accelerated Reader Program is continued so that students are given the opportunity to use this resource to assist them with a more individualized method of self-improvement on reading comprehension. It is recommended that all ABC Elementary School teachers meet to share ideas of how to better monitor Accelerated Reader quizzes and ZPD levels. Those teachers who have the higher gains on STAR and MAP should share the strategies that they use in their classrooms. It is recommended that a Parent Night be held to explain to parents the importance of this program in helping their child improve in reading overall.

As a result of this evaluation, it is our recommendation that the Accelerated Reader program be continued at ABC Elementary School. Teachers whose test scores are extremely high need to assist other teachers in bringing their students up to the standards as well.

References
(2012). Renaissance learning. Retrieved from http://www.renlearn.com/. (2012). School data reports ABC Elementary. Retrieved from School Data Clerk and Testing Coordinator.*

*Name of School has been changed.

Appendix
Appendix A ABC Elementary School Staff Survey Your answers to the following questions will support our analysis of the Accelerated Reader and Star Reader programs as they are used in our curriculum at ABC Elementary School. 1. How many years of teaching experience do you have? __________ 2. What is the highest degree you have received? Please circle one answer. a. Bachelor Degree b. Bachelor Degree, plus___hours c. Masters Degree d. Masters Degree, plus___credits e. Ph.D. 3. What grade level(s) do you presently teach? _________ 4. My primary teaching assignment for this year is (are): a. Elementary b. Elementary, Inclusion class c. Special Education d. Other (please list)_____________________________ _________________________________________ 5. In which areas do you hold certification? a. Elementary b. Secondary c. Special Education d. Reading e. Science f. Language Arts g. Mathematics h. Fine Arts i. Music j. Technology k. Other_____________________________________ _________________________________________ 6. Is (are) your primary teaching assignment(s) in an area where you hold Certification? Yes No

Survey

1. How comfortable do you feel with using Accelerated Reader? Very comfortable Somewhat comfortable Not comfortable

2. Do you know how to set goals for students? Yes No 3. Do all students know their ZPD Range? Yes No 4. Do you know how to review the reports, use all reports, and monitor progress? Yes No 5. How is AR embedded into the required 30 minute daily routine of your class? 6. What do you use to monitor students' reading (i.e. reading logs)? 7. How often do your students take AR quizzes? 3 or more times a week Once a week Every other week Whenever we have time Never Other, please specify 8. Do you use the AR reports to modify your instruction?

Yes No Sometimes Other, please specify 9. If so how? 10. Do you require different genres of AR reading? Yes No Sometimes Other, please specify 11. How are you incorporating informational text and other content areas with AR? Explain.