EVALUATION OF SFX LAPTOP PROGRAM

Evaluation of St. Francis Xavier Junior High Laptop Program within the Language Arts Curriculum By: Lauren Pinto EDTECH 505-4172 Submitted to Dr. Thompson on July 25, 2012

EVALUATION OF SFX LAPTOP PROGRAM

Table of Contents Executive Summary Purposes of Evaluation Background Information Evaluation Method Results Discussion Project Cost Appendices 3 3 4 6 7 9 10 11

EVALUATION OF SFX LAPTOP PROGRAM

Executive Summary ! This is an evaluation of the St. Francis Xavier School junior high school laptop program. The program was instituted during the 2011-2012 school year with the hopes of aiding the students while marketing the school as a technology school. The laptops were placed in Ms. Lauren Pintos classroom, and were used for a variety of tasks, activities, projects, and assessments. Prior to the laptops being instituted, a formal evaluation plan was not created. As a result, objectives were determined by the technology planning committee in hindsight, and those objectives dealt with: depth of student work, proficiency with GoogleDocs and GoogleDrawing, laptop usage for classroom purposes, easement of day-to-day tasks, reduction of paper use, and alleviating the student attrition rate. The evaluation was conducted between June 4, 2012 and July 25, 2012. During that time, the evaluator developed a student survey which she posted online for one week and a half. All junior high students were invited to respond, but only eighteen completed the form. Ms. Pinto also reflected on using the laptops, and her responses were compared to the students to understand how the laptops worked within her classroom. The data revealed that the laptop program was mostly effective; several of the goals were met in full while others give St. Francis Xavier School room to grow. The goals which were wholly achieved are: usage of laptops, paper reduction, and easement of day-to-day tasks. Those goals which offer room for improvements are: depth of student work, proficiency with GoogleDocs and GoogleDrawing, and alleviating the attrition rate. As a result, the evaluator has recommended several strategies to the St. Francis Xavier stakeholders to improve their laptop program. Purposes of Evaluation General Purpose When the St. Francis Xavier junior high school laptop program was first instituted, a formal plan for implementation and evaluation was not created. As a result, it is necessary that the school uses this evaluation process to understand its investment so as to see how the devices impacted student work. Also, as the school moves forward and attempts to brand itself as a technology school, it is imperative to determine if this program is helping to meet this objective. Central Questions to be Answered Does student work show definitions, examples, reflection and/or application? Do students report that their work shows definitions, examples, reflection and/or application? Can students work collaboratively and proficiently with GoogleDocs and Google Drawing? Do students report working collaboratively and proficiently with GoogleDocs and GoogleDrawing? Are the laptops used for note-taking, revision, creation of assignments, and assessment?
3

EVALUATION OF SFX LAPTOP PROGRAM

What do the students report using the laptops for? Are day-to-day tasks eased (e.g., note-taking, revision)? Is paper use reduced and/or saved? Has the laptop program recruited new students and kept old students? Do the students report the laptops being a big factor in staying at the school? Impact of Evaluation Of the stakeholders, the one who stands to benefit most from this evaluation is the classroom teacher. This is the case because she is the one who works most closely with the devices and the students. Also, the devices are meant to support the junior highs constructivist approach to the curriculum, so this evaluation will let her know whether or not this is happening by determining how the students are using the laptops. Background Information Program Origin Since St. Francis Xavier Schools founding in 1956, the school has built a reputation within the Burbank, California community for providing an outstanding Catholic education. For many years, the school turned away students on the basis that their classes were full and could not accommodate the extra interest. Since the 2007 recession, however, the school has found itself in the position of losing students to the public sector or to schools which offer different resources or extra-curricular activities. As a result, St. Francis Xavier purchased and implemented educational technology. Starting in the 2008-2009 school year, the school purchased SmartBoards for each classroom, and since then, has purchased additional hardware to work in conjunction with the SmartBoards. By doing so, St. Francis Xavier attempted to brand themselves as the technology school since other schools in the area (both public and parochial) lacked this type of hardware. However, the purchase of these devices did not seem to make a difference with enrollment, so in an effort to market themselves further, the school began to look into laptops for 1:1 student use. The school hoped that capitalizing on the ubiquity of technology would draw in new students while simultaneously helping the school keep their reputation for providing a quality education. So, funds were allocated for the purchase of a classroom set of laptops. St. Francis Xavier has one class per grade level (K-8), with the schools junior high program (grades 6-8) rotating the three grades among three teachers. Because this rotation provided access to the most amount of students, the technology planning committee chose the junior high program and Ms. Lauren Pinto, the junior high language arts teacher, to use the computers. Thirty-eight laptops along with accompanying software and hardware were purchased in July 2011 for approximately $25,000, and were implemented for the 2011-2012 school year. Program Goals In an effort to determine and understand the programs goals, the evaluator spoke with St. Francis Xaviers technology planning committee, which is made up of the schools accounts

EVALUATION OF SFX LAPTOP PROGRAM

manager, vice principal/fifth grade teacher, and Ms. Pinto. Together, the technology planning committee identified the following six objectives: 1. Student work shows definitions, examples, reflection and/or application 2. Students work collaboratively and proficiently with GoogleDocs and Google Drawing 3. Laptops are used for note-taking, revision, creation of assignments, and assessment 4. Day-to-day tasks are eased (e.g., note-taking, revision) 5. Paper use is reduced and/or saved 6. Laptop program recruits new students and keeps old students Previous Programs While St. Francis Xavier has purchased educational technology in the past, the school had no prior experience in purchasing and/or implementing a 1:1 laptop program. Yet, in purchasing and using other educational technology, the school learned that student devices need to draw upon prior experience (whether a device is used at home or at school), small enough to be handled on student desks, easy to use, able to be kept in a centralized place, and interactive. People Involved in Program The following is a list and description of all those involved in the junior high laptop program at St. Francis Xavier School. Mrs. Rita Recker- St. Francis Xavier Accounts Manager. Rita Recker has been working at St. Francis Xavier for fifteen years. She is in charge of allocating funds for the school and the parish; this includes but is not limited to: parish activities, textbook budgeting, classroom budgeting, building repairs, office supplies, and office machinery repairs. For the schools laptop program, Mrs. Recker was in charge of budgeting and purchasing thirty-eight computer units, a charging cart, and a single license of Smart ClassroomSuite software. Mrs. Teresa Smith- Fifth Grade Teacher and Vice Principal. Teresa Smith has been working as the St. Francis Xavier fifth-grade teacher for thirteen years. Ten years into her time at the school, she had the privilege of being appointed Vice Principal. As part of the planning committee, Mrs. Smith provided both an administrative and teacher perspective. Ms. Lauren Pinto- Junior High Language Arts Teacher. Ms. Pinto has been working at St. Francis Xavier for five years. She is comfortable working with technology, and because she is pursuing a graduate degree in educational technology, she was the perfect teacher to use the laptops with the junior high students. However, Ms. Pinto is not an expert with computers, and when issues arose, it was her responsibility to seek help and repair any issues. Junior High Students. Ninety individuals comprise the junior high student population. The students come from various socio-economic statuses (ranging from blue collar to affluent) and have various technical abilities (from competent to advanced). Characteristics of the Program Devices. Thirty-eight student laptops were purchased, with three students across the three grades sharing one device. This resulted in thirty-four laptops which were used by students and four extra devices for incidentals. All laptops are clamshell-tablets with a ruggedized shell; they could be used as a traditional laptop or rotate to become a touch-screen tablet. Each unit has a
5

EVALUATION OF SFX LAPTOP PROGRAM

webcam, enabled wifi, two USB ports, and came equipped with: Windows 7, Microsoft Word (Starter Edition), Microsoft Excel (Starter Edition), and ArtRage 2 (drawing software). With the purchase of Smarts ClassroomSuite, all laptops are installed with a host of Smart products which include: Notebook (interactive lesson software), Sync (remote desktop classroom management software for teacher use), and Response (assessment software). The battery life for each unit is approximately six hours, resulting in the need to charge the computers mid-day. In order to do this, a charging cart was purchased. This cart simultaneously charges thirty-six laptops, and at the close of each school day, the laptops are securely stored within it. Internet Infrastructure. St. Francis Xavier has a wireless Internet connection which allow the laptops to be online. The wireless access point is housed approximately thirty feet away from Ms. Pintos classroom in a room separated by cement walls. Unfortunately, the Internet is not always reliable, resulting in a sometimes poor Internet connection which went on to cause problems for students working online. SmartSync offered the ability to filter certain web content, but Ms. Pinto reported not using it often. Thus, issues surrounding website appropriateness arose. Accounts. Each student was given their own user account on their assigned device under the main administrative account. Students were given permission to customize their desktop pictures, mouse pointer, and screen savers to reflect their personality. No student was given permission to enter anothers account. Google. Each student was given a free Google for Education account. This granted the students access to GMail and GoogleDocs while giving Ms. Pinto administrative control over the accounts. Ms. Pinto had her students use these Google accounts to create presentations, and many of them chose to use GoogleDocs as their word processing program. Edmodo. Ms. Pinto employed the use of Edmodo.com, a private microblogging site built for educational purposes. Each student created an Edmodo account and joined their classmates in an online community for their grade level. Ms. Pinto used this site to distribute handouts and assessments as well as collect assignments. Students were given a non-curricular group which let them communicate with each other about non-school topics. Evaluation Method Participants The participants in this evaluation came from junior high population of ninety students. Because this evaluation occurred over summer vacation between the 2011-2012 and 2012-2013 school years, a judgement sample was taken. The survey was distributed via Edmodo.com, and those students who visited the site between June 22, 2012 and July 2, 2012 were encouraged to take the survey. A total of eighteen students completed the survey, six of whom just completed their sixth-grade year and twelve of whom just completed their seventh-grade year. Procedures Due to summer vacation, the laptop program was not in use during the time which this evaluation took place. A survey was sent out to participants asking them to reflect on their experience using the laptops. Using Edmodo.com, student work from those who answered the
6

EVALUATION OF SFX LAPTOP PROGRAM

survey was analyzed for definitions, examples, reflection, and/or application; the amount of work submitted via Edmodo was also tabulated to determine an approximation as to how much paper was saved. Data Sources Data was collected regarding the objectives of the program: how the students used the laptops, students skill level with GoogleDocs, ease of day-to-day tasks, whether paper was saved, and whether the laptops impact student return rate. General comments and information was amassed as well, providing the evaluator with more information about what other skills were acquired by working with laptops. These comments are intended to provide the evaluator with a holistic understanding of the program, aside from looking at the stated objectives. A permanent link to the survey administered to students as well as a permanent link to the results can be found in Appendix A. Results Student Work Two assignments completed by sixth graders and two assignments completed by seventh graders were Finish this statement, "When used as part of analyzed. class, the laptops let me..." Sixth-grade Assignments. The sixth-grade assignments which were analyzed were: Mr. Stick Presents Conjunctions and Interjections slides. Both assignments contain either definitions, examples, and/or application of a concept. Samples of student work can be found in Appendix B. Seventh-grade Assignments. The seventh-grade assignments which were analyzed are: School Newspaper Article and Thank You Cards. Both assignments contain either definitions, examples, and/or application of a Figure 1.1 concept. Samples of student work can be found in Appendix C. Student Responses. Students were asked if the laptops allowed them to add definitions, examples, reflection, and/or application to their assignments. Seventeen participants (94%) stated that they were able to add definitions to their work, sixteen (88%) stated that they could make something based on what they learned, thirteen (72%) said that they could add an example, and nine (50%) stated that they were able to think about what they learned (Figure 1.1). Student Use of Laptops All participants reported using the laptops for note-taking, presentation, and quizzes/test. Between fifteen and seventeen students stated that the laptops were also used for: drafting, editing, and revising assignments; adding pictures to notes or assignments; drawing through

EVALUATION OF SFX LAPTOP PROGRAM

either ArtRage 2 or GoogleDrawing; playing games which were part of class; and web searching that was part of class. Student Skill Level with GoogleDocs Students were given a list of eight skills related to GoogleDocs such as creating a new file, sharing the Doc with others, adding a picture, and changing the font size. They were asked to rate each skill, and then count how many skills they are good or great at. Of the eighteen responses, thirteen students (72%) state that they are competent at six or more skills (Figure 1.2). Using the skills above, how many of them did you rate as good or great?

Figure 1.2 Day-to-Day Tasks ! All participants stated that the laptops eased day-to-day tasks, citing, among other reasons, that they were able to type notes and save them digitally, look up information faster, and start assignments at school. Saving of Paper All participants stated that paper was saved because notes, assignments, and assessments were mostly done on the computers. Forty-eight total tests and 105 total assignments were turned in via Edmodo.com across all three grade levels. Each test and assignment was minimum one page of paper, which means that at least 18,090 sheets of paper were saved. Attrition Rate If you are coming back next year, are the laptops the main reason for staying? If you are not coming back next year, are the laptops the main reason that would make you want to stay?

Figure 1.3

Figure 1.4

Students were asked two questions which were meant to uncover how well the laptops worked as a marketing tool. Seventy-eight percent stated that the laptops are not the main reason for returning to the school (Figure 1.3), and the participants were split in half as to whether the laptops would make them want to stay if they were to leave the school (Figure 1.4).

EVALUATION OF SFX LAPTOP PROGRAM

Discussion ! The purpose of evaluating the laptop program was to determine its effectiveness against the six stated objectives identified by the technology planning committee. The data and evaluation show that the program was mostly effective. Further discussion follows below. Objective 1: Student work shows definitions, examples, reflection and/or application Student work consistently contains definitions, examples, reflection, and/or application, and students report that their works has these elements. For this reason, this objective is deemed as mostly effective. Reflection is an element that is missing from a lot of student work, so it is recommended that Ms. Pinto add more ways for students to reflect in their work. This can come in the form of a writing prompt, reflection question, or progress points wherein students look at their changes throughout the course of the year. Objective 2: Students work collaboratively and proficiently with GoogleDocs and Google Drawing ! Both students and Ms. Pinto report that most are able to perform the basic functions of GoogleDocs and GoogleDrawing. For this reason, this objective is deemed as effective. It is recommended that Ms. Pinto spends more time at the start of the year acquainting the students with the functions of the GoogleDocs and GoogleDrawing; this can be done via whole-group or small-group mini-lessons. Objective 3: Laptops are used for note-taking, revision, creation of assignments, and assessment ! Both students and Ms. Pinto reported that laptops were used in the capacities listed above, however, some students did report using the laptops for recreational purposes like online shopping, Google searching, or games not related to class. For this reason, this objective is deemed as effective. It is suggested that Ms. Pinto continues to use the laptops as she has been, while making sure to use the Smart Sync remote desktop software to better monitor and manage students who are not on task. Objective 4: Day-to-day tasks are eased (e.g., note-taking, revision) ! The laptops allowed students to keep better track of their work, access their work at home, submit assignments outside of school hours, finish work quicker, and streamline materials. For this reason, this objective is deemed at highly effective. It is suggested that Ms. Pinto continues to use the laptops in this way. Objective 5: Paper use is reduced and/or saved At least 18,090 sheets of paper were used, and students state that they used paper for only selected assignments within Ms. Pintos class. As a result, this objective is deemed at highly effective. It is recommended that Ms. Pinto continues to digitize assignments, and strive toward making the classroom completely paperless. Objective 6: Laptop program recruits new students and keeps old students
9

EVALUATION OF SFX LAPTOP PROGRAM

The results show that the laptops did not provide a significant incentive for staying at the school. Moreover, the school only welcomed two new students to the junior high for the 2012-2013 school year while losing nine. It is because of this that this objective is deemed ineffective. It is recommended that the school not attempt to bolster the laptops (or other educational technology) as the sole reason for having new families attend the school, but rather highlight technology as a learning addition to the methods the teachers use. Project Cost The budget below is inclusive of all services rendered and costs related to the evaluation of the St. Francis Xavier laptop program. The evaluator salary is the largest value, while the others are reflective of the day-to-day necessities for completing this evaluation. The evaluation took thirty days, and during that time, the evaluator spoke with stakeholders, developed and distributed the student survey, collected survey results, analyzed data, and wrote the evaluation report. Because speaking with stakeholders took only three days, talk and text time was kept to a minimum. However, since the surveys were distributed and collected via GoogleForms, Internet access was required for the full time. Lastly, since this evaluation took place over the schools summer vacation, the evaluator was able to speak with stakeholders, collect and analyze data, and write the evaluation report from from home during the thirty days; no cost related to travel was accrued. Personnel (Includes per diem) Name Evaluator (Lauren Pinto) Communication Area Talk and Text Internet Travel Reason Speaking with Rita Recker and Teresa Smith Data collection and analysis Writing of evaluation report None None None $0 $0 $0 TOTAL: $15,085
10

Salary 30 days @ $500 $15,000

Cost

Number of Days 3 days @ $100/month 30 days @ $75/month $10 $75

Total Cost

EVALUATION OF SFX LAPTOP PROGRAM

Appendices

11

EVALUATION OF SFX LAPTOP PROGRAM

Appendix A: Student Survey and Responses Student Survey The student surveys can be found at the following link: http://tinyurl.com/88p8hdz Student Responses
The student responses can be found at the following link: http://tinyurl.com/6r3o4s5

12

EVALUATION OF SFX LAPTOP PROGRAM

Appendix B: Sixth-Grade Student Work Mr. Stick Presents: Conjunctions

13

EVALUATION OF SFX LAPTOP PROGRAM

Interjection Slides

14

EVALUATION OF SFX LAPTOP PROGRAM

15

EVALUATION OF SFX LAPTOP PROGRAM

Appendix C: Seventh-Grade Student Work School Newspaper

16

EVALUATION OF SFX LAPTOP PROGRAM

17

EVALUATION OF SFX LAPTOP PROGRAM

18

EVALUATION OF SFX LAPTOP PROGRAM

Thank You Cards

19

EVALUATION OF SFX LAPTOP PROGRAM

20