Running head: EVALUATION REPORT 1

Evaluation Report: Texas Southern University Summer of Success Program Evaluation

Heather Cantu, Kailey Studhalter, and Cortni Williams

Texas A&M University

EHRD 618: Evaluation Models in Human Resource Development

Dr. Khalil Dirani

August 3, 2019
EVALUATION REPORT

2
Texas Southern University Summer of Success Program Evaluation

Executive Summary

During the Summer 2019 academic semester, as part of our EHRD 618 (Evaluation

Models in Human Resource Development) course, our team worked with Texas Southern

University (TSU) to evaluate its Summer of Success (SOS) Program. The following document

communicates the process of analyzing the students' relation to the successes of the program, as

they specifically relate to the students whom are incoming freshmen enrolled in the SOS

program at the Houston location. Through the process of surveying the students of the TSU SOS

Program, our team was able to identify these key findings.

The TSU SOS Program appears to be successful in achieving success in the key question

we evaluated which is: the program familiarizes students with the university-setting so that they

can be successful students and enroll in courses for the Fall semester (i.e., become official TSU

students). We achieved a one-hundred percent response rate from our sample group, so we feel

comfortable in generalizing these results to the SOS Program as a whole. Overall, nineteen of the

respondents surveyed revealed that they feel the SOS Program increased their comfort level with

college life. Eleven of the respondents believe that good study habits are the most important

factor in being a successful college student, which is one of the key mindsets that the program

hopes to instill in its participants.

We recommend that TSU continue to follow the same protocols in administering the SOS

Program to applicable freshmen, as we evaluate the program to be successful. We are interested

in graduation rates of participating students since this program has only been in progress for two

years. We think it will be important for TSU to re-evaluate the program in two years’ time to

check the graduation rates of these students and adjust the program accordingly.
EVALUATION REPORT

3
Project Description

For the purpose of this assignment, Heather Cantu, Kailey Studhalter and Cortni

Williams have chosen to work with TSU based in Houston, Texas. TSU has,

“an impressive array of more than 100 undergraduate and graduate programs and

concentrations, a diverse faculty, 80-plus student organizations, and an extensive alumni

network comprised of educators, entrepreneurs, public servants, lawyers, pilots, artists,

and more, many of whom are change agents on the local, national and international

stage. Texas Southern University is one of the nation’s largest historically black

universities” (About Us Texas Southern University, 2019).

The purpose of this evaluation project is to evaluate the university’s SOS Program, a pre-college

academic program at TSU. As Oesterreich (2000) notes, “the technical components of pre-

college programs cover the structures, strategies, and knowledge necessary to prepare students

for admission, enrollment, and graduation from college” (p. 2).

TSU’s SOS Program is a five-week residential session in the heart of Houston, Texas.

Students are admitted to this program because they were not admitted to TSU. Still, TSU would

like students to have the option to work their way into the university. The SOS Program is

designed to (1) improve students’ academic preparedness for college; (2) assist them in preparing

for success in a college environment; and (3) give them a unique ‘early start’ opportunity to

begin satisfying college academic requirements (James, 2019, p. 2). Our goal for the project is to

work with TSU and their students to determine: overall, is the program effective in familiarizing

students with the university-setting so that they can be successful and enroll in the Fall semester?

Overview of the Organization:

EVALUATION REPORT

4
During the first three weeks of the program, students will participate in two college-credit

classes, two academic labs, a freshman seminar course, and programming to introduce them to

college life. The academic labs are placed in the student’s schedule based on their test scores.

Those students who scored poorly on the math section of the standardized test they took, will be

placed in a math academic lab. Those students who scored poorly on the reading section of the

standardized test they took, will be placed in a reading academic lab, and so on.

The freshman seminar course will show them how to navigate through their student

portal, how to manage their time in college, how to study, as well as many other lessons needed

in order to be successful in college. Programming will include getting involved with the

Department of Student Life so make sure that they are getting acclimated socially to TSU. After

week three, students and parents will receive a progress report to show their current standing.

The last two weeks of the program are dedicated to those students who have completed their

classes by earning a 2.0 grade-point average (GPA) or higher with satisfactory conduct in the

program (James, 2019, p. 2). These students will be able to participate in a ceremony to celebrate

their success and register for the Fall semester.

Key Objectives of the Program:

1. By the end of the summer term, SOS students who have successfully earned a 2.0 or

higher GPA will develop the necessary requirements to gain full admission to TSU.

2. Instructors and advisors for the SOS Program will show students commitment through

engagement and involvement at the University with student organizations, on-campus

activities, and learning communities.

3. Students will enhance their skills by utilizing the resources available through the tutoring

service called the ‘Tiger Hub’.

EVALUATION REPORT

5
4. Instructors and advisors will serve as mentors for students to make them more

comfortable with the college environment

Key Question/Purpose of the Evaluation:

The key question to be evaluated in this study is:

1. Is the program effective in familiarizing students with the university-setting so that they

can be successful and enroll in the Fall semester?

The key purpose of this evaluation is to determine the magnitude of success of the SOS Program

at TSU. In making this determination, the SOS Program can be evaluated for worthiness of

continuation. In other words, the university could make the decision of whether or not the

program should continue as is, or if changes should be made in order to better serve the targeted

student population.
EVALUATION REPORT 6

Logic Model

Logic Model for the Texas Southern University (TSU) Summer of Success Program

Resources/Inputs Students will invest:

● Time
● Money
● Knowledge
Developers will invest:
● Faculty/Staff Time
● Expertise
● Materials
● Technology
● Partners
● Planning Time

Activities The activities which the student will engage in to meet program objectives:
● Learning Labs which focus on their deficiency whether that be in Math or English
● Academic lectures that are on their degree plan in order to be considered in ‘good standing’ with the
university
● College 101 course which helps them get acclimated in the university (some topics in this course include
time management, study skills, and note-taking)
● Tutoring Services
● Engaging social activities that allow students to get to know others
● One-on-one meetings with advisers and instructors, which breaks the fear of communication with older
adults

Outputs The expected outcomes of the program include:

● Increased academic preparation for college
● The ability of new students to begin satisfying college academic requirements early on in their
undergraduate careers
● Improved GPAs of participating students.
EVALUATION REPORT 7

Outcomes The expected outcomes of this program include:

● Graduate from the program with six college credit hours.
● Be better prepared for the fall semester due to having prior experience on campus, possessing knowledge
about Blackboard, having the ability to effectively communicate with staff and faculty, etc.
● Will be enrolled as a full time student

Impact The overall impact of the Summer of Success Program at TSU is:
● Increased Freshman enrollment
● Increased likelihood of graduation with a bachelor’s degree within four years among program participants
(i.e., improvement of graduation rates at TSU)
 EVALUATION REPORT 8

Methodology

As graduate students, the best way for us to develop experience is through “…case

studies, exemplars from practice, role-playing, or critical incidents – as a common means of

engaging students in understanding concepts, principles, or theories” (Merriam, Caffarella, &

Baumgartner, 2007, p. 210). The predominant purpose for our evaluation was to serve as a case

study imparting valuable experience in preparation for us to develop our abilities to help identify

indicators and best practices for evaluation. We utilized the practices initially recommended by

our professor, Dr. Khalil Dirani.

To carry-out our data-gathering, our team chose to use quantitative questions to evaluate

our sample group. Our evaluation is considered a quantitative case study. The questions

presented to the students was based on our primary evaluative questions listed in the Project

Description. The goal for the project was to work with TSU and their students to determine the

answer to this question.

To complete this task, our team has developed quantitative evaluative questions which

we will distribute to students participating in TSU SOS Program via Google Survey. Please see

an example of our questions below (the remainder of questions can be found in Appendix A).

Survey Questions Answer Choices

 EVALUATION REPORT 9

How often do you currently attend tutoring a) Not at all

per week, on average? b) 1 to 2 times per week
c) 3 to 4 times per week
d) 5 or more times per week

How many hours per day do you spend on a) None

schoolwork, on average? b) 1 to 2 hours
c) 3 to 4 hours
d) 5 or more hours
Table 1. Survey Question Examples

The method we used for data collection was Google Survey, and we disseminated our

quantitative questionnaires to the sample group using this program. Google Survey’s

functionality allows for data to be automatically gathered and converted into pie-charts, which

our team then analyzed for trends.

At the beginning of the summer program, students are required to take two college-level

classes, two labs (math and reading), as well as a college success course. In this college success

course, they learn about the tools they can use to be successful. Students are given a survey at

their second meeting in the college success course. This survey will represent quantitative

research questions. We will analyze the data collected for trends in students’ beliefs and

progress. Our data is composed of quantitative elements.

Results

Collected and Analyzed Data:

Through Google Survey, we collected data surrounding eight survey questions from the

sample of participants. Our sample included twenty SOS Program students at TSU. To analyze

the data, our team utilized the pie-charts that Google survey provided. The most common

responses to our questions are shown in the table below:

Question Answer Choices

 EVALUATION REPORT
How often do you currently attend tutoring
per week?
How many hours per day do you study and do
homework?
What is your current GPA?
Please share your expected GPA for the end
of this program.
What is the most important factor in being
successful in college from the list below?
What is the most notable negative behavior in
people who are not successful in college from
the list below?
Do you feel as though the program increased
your comfort level with college life?
In what way does studying most influence the
success of college students?
Table 2. Survey Questions and Responses
Reported Findings
The survey provided us with an understanding that majority of the students participating
in the SOS Program have overall become more comfortable with the idea of college. As
10
Not at all (7 participants)
1-2 times per week (11 participants)
3-4 times per week (1 participant)
5 or more times per week (1 participant)
Not at all (1 participant)
1-2 hours (13 participants)
3-4 hours (5 participants)
5 or more hours (1 participant)
Less than 2.0 (2 participants)
2.0 to 2.5 (3 participants)
2.5 to 3.0 (9 participants)
Greater than 3.5 (6 participants)
Less than 2.0 (1 participant)
2.0 to 2.5 (2 participants)
2.5 to 3.0 (5 participants)
Greater than 3.5 (12 participants)
Good study habits (11 of participants)
Being involved (3 participants)
Internship opportunities (2 participants)
Being a good writer (1 participant)
Networking (3 participants)
Poor study habits (11 of participants)
Isolating oneself (4 participants)
Being a poor writer (2 participants)
Not being involved (3 participants)
Yes (19 participants)
No (1 participant)
Enhances dedication (10 participants)
Enhances long-term memory regarding topics
(8 participants)
Prepares students for the workforce (2
participants)
 EVALUATION REPORT 11

Baumann, Miller & Sorenson (2005) noted, the primary objective of these type of programs is to

“help students develop academic skills that will improve their likelihood of attending and

succeeding in college” (p. 1). Of the twenty students who completed our survey, nineteen

responded ‘yes’ to the question regarding, do you feel as though the program increased your

comfort level with college life? Interestingly, most of the most frequently chosen responses had

a fifty-percent or greater selection rate. We believe that this high-percentage lends itself to the

high internal validity of our evaluation. In other words, “... the observed differences on the

dependent variable are directly related to the independent variable, and not due to some other

unintended variable” (Fraenkel, Wallen, & Hyun, 2019, p. 161). In this same vein, we had a one-

hundred percent response rate from the sought-after participants, which increases external

validity. “The best sampling plan is of no value if information is missing on a sizable portion of

the initial sample” (Fraenkel et al., 2019, p. 104). That is, we feel comfortable in generalizing our

findings to SOS Program students at large. Additionally, the question regarding whether the SOS

Program at TSU increased the students’ comfort level with college life had an overwhelmingly

positive response rate (nineteen selected ‘yes’) from participants. The students are the ones that

know best about how they are feeling regarding their studies, so we believe this is an important

fact to note. Since nineteen of the students feel more comfortable with college life after being

enrolled in the SOS Program, we believe that this data lends credence to the fact that the program

is successful at its goals overall.

Discuss findings/results and connect them to the purpose/evaluation questions

Interpretation of Results
 EVALUATION REPORT 12

Overall, it appears the participants in the SOS Program at TSU feel as though the

program is sufficiently acclimating them to college life. There appears to be a trend in the data in

that the students believe good study habits are key in being a successful college student.

Additionally, most SOS Program students believe they will achieve a 3.5 GPA or higher at the

close of the program, which means that these students will successfully complete the program

and be able to register for the Fall semester. Finally, participants believe that poor study habits

are the main reason that students are unsuccessful in college. It appears the SOS Program at TSU

has instituted the program’s goal mindset in the participating students in that they have good

study habits (they study at least an hour or two per day), and the students believe that this will

push them toward success in college. The students are also taking advantage of tutoring

opportunities, which is a great resource for these students to have access to. Ultimately, our team

believes that the SOS Program is currently successful at TSU due to these findings.

Implications

The present evaluation implies that the SOS Program at TSU allows for non-accepted

TSU freshmen to increase their comfort level with college life, learn good study habits, and learn

the value of being engaged in their college environment. We are pleased with the positive results

of our evaluation and are interested to see the results of how many of these students will graduate

with their bachelor’s degree within four years. If current patterns continue, we believe that a

significant number of these students will earn their degrees within this timeframe.

Recommendations

Our team recommends that TSU continue to follow the same protocols in administering

the SOS Program, as we evaluate it to be successful. The university has given non-accepted

students the opportunity to grow into successful college students and have a chance at furthering
 EVALUATION REPORT 13

their education through this program. This program has only been in effect for two years, so we

believe it will be important for TSU to evaluate how many of the participating students graduate

with a bachelor’s degree within four years. This data will be key in determining the true success

of the program. In other words, if these students adhere to the goals set forth in the SOS Program

and continue to have good study habits and be involved with TSU, our team believes that they

will graduate within four years. However, if there is a significant drop-off in the number of

students that graduate, TSU will need to determine why this is the case and work to improve the

program. Additionally, our team interested in the types of degrees these students will achieve.

Will they stick to their original plan or will they change majors’ multiple times? We believe this

is also important for TSU to evaluate as discussing the field of study may be important to the

success of students that participate in the SOS Program. If they are armed with the knowledge of

what field they will be successful in going into school, they might be more apt to adhere to the

program as they will be interested in the subject matter and see the results of their study habits

unfold before them.

Conclusions

Throughout this process, our team has learned and applied ourselves to the area of

Organizational Evaluation. We worked with Texas Southern University and Summer of Success

Program to provide them with concrete recommendations so that they can increase the success of

the program. From survey responses that we received, we believe that we have created credible

recommendations for the program.

References

About Us. (2019). Retrieved from http://www.tsu.edu/about/index.html

 EVALUATION REPORT 14

Baumann, S., Miller, P., & Sorenson, S. (2005). The role of culture in college preparation

programs: A review of the research literature. Preparing for college: Nine elements of

effective outreach, 13.

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2019). How to design and evaluate research in

education (10th ed.). New York, NY: McGraw-Hill Education.

James, C. (2019). Summer of Success Academic Academy. Houston, TX.

Merriam, S. B., Caffarella, R. S., & Baumgartner, L. M. (2007). Learning in Adulthood:

Comprehensive Guide (3rd ed.). San Francisco, CA: Jossey-Bass

Oesterreich, H. (2000). The Technical, Cultural, and Political Factors in College Preparation

Programs for Urban and Minority Youth. ERIC Digest Number 158.

Appendices

Appendix A- Survey Questions

 EVALUATION REPORT 15

Survey Questions Answer Choices

How often do you currently attend tutoring a) Not at all

per week, on average? b) 1 to 2 times per week
c) 3 to 4 times per week
d) 5 or more times per week

How many hours per day do you spend on a) None

schoolwork, on average? b) 1 to 2 hours
c) 3 to 4 hours
d) 5 or more hours

What is your current GPA? a) Less than 2.0

b) 2.0 to 2.5
c) 2.5 to 3.0
d) 3.0 to 3.5
e) Greater than 3.5

What do you expect your GPA to be at the a) Less than 2.0

end of this program? b) 2.0 to 2.5
c) 2.5 to 3.0
d) 3.0 to 3.5
e) Greater than 3.5

What is the most important factor in being a) Good study habits

successful in college from the choices below? b) Being involved (in clubs,
organizations, etc.)
c) Internship opportunities
d) Being a good writer
e) Networking

What is the most notable negative behavior of a) Poor study habits

people who are not successful in college from b) Isolating oneself
the choices below? c) Being a poor writer
d) Not being involved (in clubs,
organizations, etc.)

Do you feel as though this program increased a) Yes

your comfort level with college life? b) No
 EVALUATION REPORT 16

In what way (from the choices below) does a) Enhances dedication

studying most influence the success of b) Enhances long-term memory
college students? regarding topics studied
c) Prepares students for the
workforce

Appendix B - Survey Results in Graphic Representation

EVALUATION REPORT 17
EVALUATION REPORT 18
EVALUATION REPORT 19