Compass Evaluation Plan - Ellington

Running head: AN EVALUATION PLAN FOR COMPASS
AN EVALUATION
An Evaluation Plan for Compass: A Peer Mentor Program PLAN FOR COMPASS:
A PEER MENTOR PROGRAM
Trebby Ellington and Alyssa D. Humbles
Loyola University ChicagoTrebby Ellington & Alyssa D. Humbles

ELPS 431: Evaluation in Higher Education
Loyola University Chicago
AN EVALUATION PLAN FOR COMPASS 2
Table of Contents
Statement of Problem and Significance ..........................................................................................4
History of SES ...................................................................................................................................4
Description of SES ............................................................................................................................6
Compass Program Details ................................................................................................................7
Stakeholders ....................................................................................................................................9
Compass Assessment Rationale.................................................................................................... 11
Logic Model Description ............................................................................................................... 12
Outcomes .......................................................................................................................... 13
Assumptions ...................................................................................................................... 15
External Factors ................................................................................................................ 15
Evaluation Approach ..................................................................................................................... 16
Evaluation Question ...................................................................................................................... 17
Reasoning for Approach................................................................................................................ 18
Quantitative Approach.................................................................................................................. 19
Quantitative Design ...................................................................................................................... 19
Comparison Group ............................................................................................................ 20
Research Design ................................................................................................................ 20
Survey Instrument and Implementation ...................................................................................... 21
Survey Details.................................................................................................................... 21
Pilot Testing....................................................................................................................... 22
Survey Delivery ................................................................................................................. 23
Statistical Analysis ......................................................................................................................... 24
Presentation of Results ..................................................................................................... 25
Qualitative Approach .................................................................................................................... 26
Interview Protocol ........................................................................................................................ 27
Pilot Testing....................................................................................................................... 29
Sampling Frame ................................................................................................................ 29
Implementation Procedures ............................................................................................. 30
Data Analysis ................................................................................................................................. 33
Limitations of Qualitative Approach ............................................................................................. 35
Presentation of Results ................................................................................................................. 35
Overall Limitations of Evaluation Plan .......................................................................................... 36
Timeline......................................................................................................................................... 37
Budget ........................................................................................................................................... 38
Next Steps ..................................................................................................................................... 39
References .................................................................................................................................... 40
Appendix A .................................................................................................................................... 42
Appendix B .................................................................................................................................... 43
Appendix C .................................................................................................................................... 45
Appendix D .................................................................................................................................... 46
Appendix E .................................................................................................................................... 47
Appendix F .................................................................................................................................... 48
Appendix G .................................................................................................................................... 51
Appendix H .................................................................................................................................... 52
Appendix I ..................................................................................................................................... 57
Appendix J ..................................................................................................................................... 60
Appendix K .................................................................................................................................... 61
Appendix L..................................................................................................................................... 62
Appendix M ................................................................................................................................... 63
Appendix N .................................................................................................................................... 64
Statement of Problem and Significance
Northwestern University (NU), a selective, private, predominantly white research
institution with an enrollment of approximately 20,000, strives to educate its students, engage
the community and enrich the Northwestern experience (SES Compass, n.d., p. 3). However,
unlike some other institutions, a number of students at Northwestern are dissatisfied with their
experiences due, in large part, to the institutions decentralization of resources and community
integration (students do not apply to NU, they apply directly to a particular school within NU) in
addition to its academic rigor. In particular, students of underrepresented and traditionally
marginalized populations (i.e. black, LGBTQIA, low-income) encounter dissatisfaction at NU at
higher levels than their traditional peers (Strategic Plan, 2016). One office created to address
some of the dissatisfaction is Student Enrichment Services (SES). SES works particularly with
low-income and/or first-generation students. Three goals that influence SESs efforts are to
demonstrate an increased sense of belonging to the Northwestern community and its
surrounding neighborhoods, develop skills to utilize campus resources to navigate communities,
organizations, and institutions, and develop strategies that enhance participation in the student
curricular and co-curricular experience (SES Compass, n.d., p. 9). To measure the success of
SES in accomplishing these goals, we will assess specifically a new initiative in the office, the
Compass peer mentor program. Prior to measuring the success of the Compass program, we will
provide an overview of the organizational structure within which it operates, its history on NUs
campus and key components of the program.
History of SES
Vice President of Student Affairs Patricia Telles-Irvin, Ed.D. was recruited to NU in
2011 to help shift the campus culture and to bridge the gap between Northwesterns strategic
themes and practices by the Division of Student Affairs and the university as a whole. To begin
this process, data were collected from staff and leadership throughout the division, both
undergraduate and graduate students, senior university officers and the board of trustees via
interviews, surveys, and a strategic planning retreat all to identify issues, opportunities and
challenges. In 2012, the Division of Student Affairs identified goals that support and align with
Northwesterns strategic themes (SES Compass, n.d., p.3). In an effort to advance social
justice, one of NUs strategic themes, and increase satisfaction in student experience, Executive
Director Dr. Lesley-Ann Brown-Henderson was hired and charged with creating the department
of Campus Inclusion and Community (CIC) in 2012. CICs goal is to collaboratively work with
the university community to foster inclusive learning environments through intentional
engagement with difference (SES Compass, n.d., p.7) to cultivate a community of respect at
NU. Over the past couple of years, Dr. Lesley-Ann Brown-Henderson has expanded the
department to include the offices of Multicultural Student Affairs (MSA), Social Justice
Education (SJE), and Student Enrichment Services (SES) within the larger context of CIC. As of
July 2015, CIC and all of its offices are overseen by the Assistant Vice President for Student
Engagement, Dr. Kelly Schaefer, under the overarching umbrella of the Division of Student
Engagement (Appendix A).
The office of SES was created in response to a focus group from spring 2013, where the
purpose of the study was to (1) explore the effect of socioeconomic background on the
Northwestern undergraduate experience; and (2) to listen to students ideas for how
Northwestern could better foster a nurturing campus climate that is inclusive and welcoming for
all students (Student Affairs Assessment & Campus Inclusion and Community, 2013, p.7).
Students whose family annual income averages about $400,000 make up 50% of NUs
population and the middle-class average family annual income for NU students is roughly
$100,000. The 14% of the NU population who identify as low-income and/or first-generation
have an estimated family annual income of $30,000 or less (K. Cockrell, personal
communication, September 24, 2016). Per these numbers and the descriptions of students
experiences in the focus group report, the elite culture at NU is not necessarily the safest nor
most inclusive space for students with these identities, which negatively impacts the satisfaction
and experiences of these students, thus highlighting the importance of the existence of the SES
office and its services.
Description of SES
Having come to fruition in November 2013, the office of SES is two years old at NU.
Through campus-wide partnerships, SES builds an inclusive Northwestern community by
engaging students and their allies with programming and dialogue around the low-income and/or
first-generation experience (SES Compass, n.d., p. 9). Up until September 2016, the office of
SES was staffed by only one person, Director Kourtney Cockrell. Recently, SES was able to
expand and hire more staff. Currently, SES consists of three full-time staff, Director Kourtney
Cockrell, Assistant Director Sharitza Rivera and Administrative Assistant Christina Smith and
two part-time staff, AmeriCorps Vista Zoey Meyer-Jens and Graduate Intern Trebby Ellington.
SES directly offers a number of resources to students who identify as low-income and/or
first-generation that includes: a loaner laptop policy where all students with need can loan out a
laptop for a quarter, a winter gear program where students receive free gear for the winter, the
Knight Family Foundation Emergency Fund used to support students with unforeseen costs
associated with emergencies (i.e. illness, travel associated with a family death, temporary
housing, etc.), and the Katz Enrichment Fund which ensures that students experiencing financial
difficulties can still fully participate in campus activities. SES also serves as a liaison between
low-income and/or first-generation students and other campus resources. An online resource
portal was created to connect students with cultural programming, financial aid, academic
support, fellowships and research, involvement, Northwestern Career Advancement, school-
based support, study abroad, health and wellness, and service and engagement (SES Compass,
n.d., p. 11).
Compass Program Details
Still very much a pilot, the Compass program (formerly known as the SES Peer Mentor
program) is in its second year of existence. As SES Compass Program CURRICULUM (n.d.)
stated:
The SES Compass program is a year-long mentorship program that connects first-year
students with trained upperclassmen mentors, all of whom come from first-generation
and/or low-income backgrounds. Through one-on-one meetings, social events, and
workshops, students learn how to successfully navigate the Northwestern experience
while developing strong relationships that will continue throughout their four years and
beyond. (p. 12).
In its first year in 2015-2016, Compass had one cohort of 16 first-year students and four mentors.
This year, 2016-2017, Compass manages two cohorts of 16 freshmen and four or five mentors
per cohort simultaneously to total 32 freshmen mentees and nine upperclassmen mentors. This
results in mentors being matched with between 3-4 mentees each. The learning outcomes
associated with the Compass program are to identify and utilize campus resources in times of
need, describe at least one student organization, activity, program or workshop in which they
engaged with this year, and express greater satisfaction with their first year experience at
Northwestern than do peers who are not part of Compass (SES Compass, n.d., p. 12).
Compass requires its mentors and mentees to participate in weekly and biweekly
sessions. On a weekly basis, all mentors and mentees within a cohort are expected to participate
in a large group cohort meeting that takes place over the dinner hour in which mentors will co-
facilitate different types of workshops pertaining to community building, campus resources, and
involvement to help the mentees get acclimated to NU. Also on a weekly basis, mentors are
required to participate in coaching sessions with SES staff to continue ongoing mentor training
about topics such as facilitation, mentorship, and NU resources. On a biweekly basis, mentors
are expected to meet individually with each of their mentees for an hour at minimum to continue
to build rapport and provide one-on-one support. In addition to these weekly and biweekly
sessions, mentors are required to plan weekend social activities that are optional for the mentees
to continue to cultivate community and enhance their sense of belonging (K. Cockrell, personal
communication, September 15, 2016).
Since mentors commit such a large amount of time to Compass, they receive a stipend of
$1,000 per quarter in which they receive $500 at the beginning and the other $500 at the end.
The Compass mentors stipend, as well as other activities related specifically to the program and
SES in general, is supported by the Querrey Fund. In addition to the mentor stipends, the
Querrey Fund supports the purchases of food for the Compass program meetings, winter gear for
the winter gear program, and most recently, the Querrey Laptop Grant Program has fully
supported the costs of 41 laptops as gifts for first-year students who identify as first-generation
and/or low-income accompanied by a four-year warranty (K. Cockrell, personal communication,
September 24, 2016).

The foundation of Compass and its curriculum is linked with Dr. Mesmin Destins
research about closing the social-class achievement gap (K. Cockrell, personal communication,
September 15, 2016). He and his colleagues studied whether an educational experience created
to help students recognize how difference matters could be used to permit first-generation
students to transition to college more effectively and overcome challenges to success that are
background specific. Utilizing a difference-education intervention, they used a panel of senior
college students at the beginning of college to share their personal stories of how their different
backgrounds shaped their college experience in positive and negative ways and different
strategies for success that consider different backgrounds contrasting first-generation and
traditional students experiences. In the standard control condition, similar stories were shared;
however, background specific information about how social class impacts success was omitted.
As a result, the difference-education intervention at the beginning of college reduced the social-
class achievement gap among first-generation and traditional college students by 63% at the
conclusion of their first-year and enhanced their psychological, academic, and social transition
and adjustment (Stephens, Hamedani & Destin, 2014). Dr. Mesmin Destins research plays an
integral role in the framework of the Compass program as Compass connects first-year low-
income and/or first-generation students with upperclassmen who also identify as low-income
and/or first-generation as a way to emphasize that people with backgrounds similar to theirs can
flourish when they use the appropriate kinds of tools and strategies that consider background
specific information.
Stakeholders
SES has a variety of people who can be impacted by the evaluation of the Compass
program. The office of SES reports to CICs Executive Director Dr. Lesley-Ann Brown-
Henderson who then reports directly to Assistant Vice President of Student Engagement, Dr.
Kelly Schaefer (Appendix A). One goal of the Division of Student Affairs is to increase the
number of students who identify as low-income and/or first-generation from 14% to 20% by
2020 (K. Cockrell, personal communication, September 15, 2016), so the services provided by
SES are imperative to learning how to appropriately serve this community before that percentage
rises by increasing the satisfaction in overall student experiences at NU. Next, donors such as
those involved with the Knight Family Emergency Fund and the Katz Enrichment Fund hold a
stake in the successes of SES. Donors want to know that the money they are investing is helping
to produce positive results, so the success of this office will help determine whether donors will
continue to donate and whether or not they will increase their donations. In order to accomplish
its goals and increase satisfaction in student experience, SES depends heavily on its campus-
wide partnerships. To assist low-income and/or first-generation students in navigating the NU
experience, SES works closely with offices that include, but are not limited to, Campus Inclusion
and Community, Residential Life, Financial Aid, Academic Advising, Counseling and
Psychological Services, and Health Services, all of whom benefit from their services being
inclusive to all students at NU.
The Compass program, in addition to the stakeholders mentioned above, have various
people who are impacted by its evaluation. Donors of the Querrey Fund want to see that the
money they are investing in the Compass program is helping to create spaces where better
community can be built and foster a better sense of belonging. The three full-time and two part-
time staff who work directly with the Compass program are also impacted by the evaluation of
the program as all are investing time facilitating workshops for the mentees, training the mentors
and developing curriculum for the program as a whole. The evaluation of the program will
illuminate what the staff are doing well and what things can be done to improve the participants
experiences in the program. Similarly, the nine mentors have a stake in the programs evaluation
as their relationships with the mentees, in large part, determine the success of the peer mentor
program. Finally, the 32 mentees who are participating in the program are directly impacted as
the successes and/or downfalls of Compass directly shape their experiences navigating the
Northwestern culture, which in turn, determines their level of satisfaction with NU.
Compass Assessment Rationale
Compass is only in its second year of existence and has not undergone any assessment
thus far, so being that the program is new, it is important to evaluate the current framework to
make improvements as it continues to grow. In its pilot year of 2015-2016, the structure of the
program was compartmentalized into three parts (since NU operates on a quarter system). The
fall quarter of the program focused heavily on community building while the winter quarter
focused on campus resources and the spring quarter focused on involvement. As a result,
creating a sense of belonging seemed to be the learning outcome that was most achieved last year
whereas identifying and utilizing campus resources and ways to get involved were much harder
to achieve. The latter is in part because the mentees grew accustomed to building an intimate
community with one another, the mentors and the SES staff that when there was an abrupt shift
from one quarter to the next where campus partners were being introduced to them, it was a
rough transition. Learning from last years Compass cohort, the framework for the 2016-2017
cohorts represents a more integrative approach where community building, campus resources,
and involvement are sequenced throughout all three quarters (K. Cockrell, personal
communication, September 15, 2016). Also, as the program concluded its first year, through
informal observation and analysis by the director, it was noted that the mentors seemed to have
gained more from their participation in the program than the mentees. Therefore, as we begin to
develop assessment tools, we will focus on assessing the mentees attainment of the learning
outcomes at the conclusion of the program.
Logic Model Description
To visually represent our understanding of the connections among the resources that
drive our program, the activities it implements and the results it is hoped to achieve, we will use
a logic model (Kellogg, 2004). The components that make up a logic model demonstrate the
relationship between planned work and intended results. The planned work components include
resources (inputs) and program activities and the intended results components include outputs,
outcomes, and impact. As Kellogg (2004) stated, The purpose of a logic model is to provide
stakeholders with a road map describing the sequence of related events connecting the need for
the planned program with the programs desired results (p. 3).
The logic model developed for the Compass program (Appendix B) illustrates the
situations that drive the justification for the proposed evaluation. As previously stated, students
who served in the mentor capacity in the 2015-2016 cohort were observed to have gleaned more
from their participation in the program than the population they set out to support. Many factors
could be attributed to why this was the case. For instance, having been a returning student as a
mentor, common pitfalls that are presented for first-year students may not have been an area of
concern for second-, third- or fourth-year students. Similarly, through their participation in the
program they were able to develop a close relationship with administrative professionals as an
extended resource and gateway to the campus. Lastly, for the mentors, their leadership role also
included formalized training that highlighted the use of campus resources and campus
involvement, thus creating a heightened sense of belonging. Understanding the benefits of the
program, tailoring of the program structure and mentor integration has been refined to increase
the probability that the mentees will, in fact, gain just as much, if not more, from their
participation in the program as the mentors.
Additionally, the logic model highlights the inputs into the Compass program. As
mentioned, there are a plethora of financial resources allocated to the Compass program by way
of donation and sponsorship. The key stakeholders who invest in the program leverage their
contributions as results driven. Moreover, the professional staff responsible for the program are
seen as an investment in multiple dimensions - the institution which finances their positions and
the staff themselves as they are investing their time and leveraging their campus relationships to
cultivate a holistic program. In addition to the outside entities responsible for funding and the
professional staff tied to the program, the mentors are also an investment. Furthermore, with
their responsibilities as a student, they self-elect to also add supplementary leadership
responsibilities that ideally will not go unrewarded. It takes a collective effort to bring a program
of this caliber to fruition.
As illustrated in the logic model, through a series of workshops, weekly and bi-weekly
meetings, deliberate time set aside to cultivate meaningful relationships with the mentors are all
activities devised to increase the satisfaction of the mentees. Mindful of the learning outcomes
set forth for the mentees in the program, seeing that some of them were not achieved in the
manner that was intended, intentional planning and program restructuring has already taken
place so that the following outcomes can be addressed in a deliberate fashion.
Outcomes
The short-, medium- and long-term goals are directly linked to the activities developed
for the intended participants. Keeping the programs specific learning outcomes in mind, as
previously mentioned, the short-term goals, meant to be attained upon immediate completion of
the program, which concludes at the end of their first year, are centered around the mentees
abilities to identify and utilize campus resources, describe at least one student organization,
activity, program or workshop in which they engaged with this year, and express greater
satisfaction with their first year experience at Northwestern than do peers who are not part of
Compass (SES Compass, n.d., p. 12). Ultimately, student satisfaction is a key area of concern,
in hopes to sustain the program. Though student satisfaction is the general concern, we recognize
that satisfaction can be interpreted in a number of ways. To be more specific and allow for more
feasible measurability, we propose new language for the programs third short-term goal that
relates to sense of belonging. This proposed language is: articulate a greater sense of inclusion
within the campus community throughout their first year experience at NU than do their peers
who are not part of Compass.
Two to three years following completion of the Compass program, the medium-term
goals are also centered on the satisfaction and sense of belonging that those who participate in
the program as mentees feel upon completion of the program. It is hoped that they would seek
leadership opportunities either within the Compass program or in other avenues at Northwestern
following their first year as a sign of increased satisfaction with their experiences. Lastly, as we
project long-term goals, with the evolution of the program, it would be ideal to identify ways that
participants in the program can maintain connection to Compass over the course of their
undergraduate trajectory at NU beyond their first-year mentee experience. Furthermore, the
medium- and long-term goals would also share a focus on growing participants knowledge and
increasing the personal, professional and academic network (social and cultural capital) that will
assist these students in positively navigating the NU community and culture following their
completion of Compass.
Assumptions
As we approach the evaluation for the Compass program, there are a series of
assumptions that are acknowledged, which will shape our quantitative and qualitative measures.
First, we must assume all of the participants share similar experiences as low-income and/or
first-generation students as it pertains to navigating the Northwestern experience. Next, we
assume that all of the participants will persist throughout the duration of the program, which lasts
a full academic year. Another assumption is that the mentor-mentee matches are sufficient for
the development and needs of the mentee throughout the course of the program. Additionally, we
assume that the delivery of the program content is facilitated in a way that is conducive to all
learning styles and is received well by all participants. Finally, considering the delivery method
of evaluation, we assume that the participants can accurately self-report their satisfaction and
their experiences as it relates to their involvement with the program.
External Factors
Considering a key component of the program is contingent on being low-income and/or
first-generation, the intersectionality of multiple identities (i.e. non-first-generation American,
gender, race) plays a pivotal role in ones experiences navigating NUs elite culture and is too
important to disregard. To coincide with the mentees participation in the program, we
acknowledge they are also students undergoing a strenuous academic rigor at a prestigious
institution that subsequently has effects on ones physical and mental health if not addressed or
supported properly. Lastly, while students are participants of the Compass program, they may
also find themselves involved in additional campus programs and organizations that assist in the
attainment of the same objectives outlined by this program.
Evaluation Approach
For the 2016-2017 school year, the Compass program has welcomed its second and third
cohort of participants. This year will present a restructured format compared to the previous
years based on informal observations. In addition to the evolution of the program structure,
Compass has doubled the size of participants, hence the simultaneous cohort model. With a
conscious effort placed on the achievement of the learning outcomes, the most appropriate
evaluation method is a formative assessment. Since Compass is only in its second year and very
much a pilot program, we want to know whether it is achieving the learning outcomes that it was
created to achieve. An outcomes-based assessment suggests gathering specific measures to
determine group assets and limitations that can then lead to action-oriented improvements. To
accomplish this, a formative assessment will be most beneficial because it requires an evaluation
of the full course of the program that will provide feedback to be used to modify, shape and
improve the program (Banta & Palomba, 2015).To better assist in the measurement of the
attainment of the learning outcomes for the mentees, a longitudinal approach will be used where
a pre and post survey design will be administered that will quantify their awareness of campus
resources, probability and actual use of said resources, as well as gauge their perceived sense of
belonging over time. With an understanding of the compartmentalized approach from the
previous year, the evaluation is intended to highlight the efforts to constantly integrate elements
of all three objectives throughout the course of the program with an intentional look at how
receptive the mentees are to the program structure as they continue throughout their year with the
Compass program.
Evaluation Question
The purpose of this evaluation is to provide substantial evidence, beyond the informal
information collected during the pilot year, to illustrate that the learning outcomes are being
achieved by the population the program set out to support. As an additional caveat, the anecdotal
data collected regarding the mentors expressed appreciation for the program will serve as a
highlight to the program. By Compass also scaling up in year two, the information collected will
hopefully advocate for additional expansion in the coming years to support the campus initiative
to increase the presence and satisfaction of this specific population by 2020. The primary
question for the programs evaluation is to qualify if the Compass program aids in overall
satisfaction of low-income and/or first-generation students who participate in the program during
their first year at NU compared to low-income and/or first-generation students who do not
participate in the program. To reiterate an earlier point, satisfaction has a number of ways in
which it can be interpreted and is difficult to measure. Based on our proposed new language, the
primary question for the programs evaluation is to qualify whether Compass aids in a greater
sense of belonging and inclusion of low-income and/or first-generation students who participate
in the program than do their peers who do not. More specifically, we want to measure whether
Compass is achieving its learning outcomes.
Furthermore, it is important to ask: does the new conceptual framework of integrated
services and topics throughout the duration of the program provide a less disruptive medium for
the participants? Through their participation, do mentees see the campus resources to be
beneficial and if so, how often do they utilize their services? At which points of the term do they
begin to leverage the relationships built with campus partners? From their participation in the
program, do the mentees feel a greater sense of belonging to the campus? Do they value the
relationships built between not only the mentors, but their fellow mentees as well? A
longitudinal, outcomes-based assessment approach will be the most advantageous as it will allow
us to determine whether the program is achieving its learning outcomes by way of observing the
program over time and comparing the mentees responses at the point of entry to their responses
at the conclusion of the program to track incremental shifts.
Reasoning for Approach
Persistence and development are greatest marked over a course of time considering both
academic and social involvement (Tinto, 1997). The Compass program initiates contact with the
mentees during the first couple of weeks in the term to encourage first-year low-income and/or
first-generation students to apply to be a mentee in the program; at which point minimal
connections have been made both curricular as well as co-curricular. The overall objective is to
measure the effectiveness of the Compass program through both quantitative and qualitative
analyses. The findings will help inform not only how to continue to improve the program, justify
the personnel and financial resources allotted to the program, but also better inform the
institution about appropriate measures to take when welcoming more students to the NU campus
who identify as low-income and/or first-generation.
In posing this approach, the strengths and areas of development for this are also
influential. First, it is important to note that corrective action has been made to the structuring of
the program, which in turn may alleviate some of the systematic concerns from the programs
first year. For this rationale, a procedural assessment may be redundant. Also to reiterate, one of
the large takeaways from the pilot program was that benefits and student gains were seen higher
from the mentors participation than the mentees. Again, to align with the original intent of the
program, it is most fitting to begin assessment with the purpose of finding out if the program is
accomplishing the goals it set out to do. The drawback to this approach is that if there are
structural components hindering the outcomes from being achieved data may be skewed based
on a number of reasons by the evaluation process being reliant on the completion of the program.
For instance, student persistence in the program can affect if the program was effective or not.
Over the course of the year that the students participate in the program, being a mentee in
Compass does not hold exclusive to their overall experience and the evaluation may not account
for outside factors that attribute to the achievement of the program objectives. By keeping this all
in mind, as the evaluation instruments are being developed, there will be intentional questions
and language that will hopefully uncover and account for these potential anomalies.
Quantitative Approach
Due to the novelty of and lack of assessment around the effectiveness of the Compass
program, we were encouraged to engage in this evaluation to see if the program is accomplishing
what it was developed to do. Additionally, it is believed that first-year students who identify as
low-income and/or first-generation feel a greater sense of belonging at NU by participating in the
Compass program than do their first-year, low-income and/or first-generation peers who do not
(SES Compass, n.d.). A quantitative analysis of the programs short-term goals will allow us to
determine whether involvement in the program positively influences the experiences of first-
year, low-income and/or first-generation students. Since the program is new, not enough time has
gone by to actually assess the medium- and long-term outcomes for the program so we will not
be analyzing the relationship between those; however, in the future, that would be appropriate to
do. In this approach, we will describe different aspects of our quantitative design, survey design,
and statistical analysis.
Quantitative Design
Remaining in congruence with the purpose of the design of Compass, the population
studied will only consist of new, first-year students who identify as low-income and/or first-
generation. Our evaluation will use a census; where the entire population serves as our sample to
be analyzed (Newcomer & Triplett, 2010). We chose to utilize a census because the population
of participants in the program is small (30 students) and all are expected to persist through their
entire first year. Furthermore, we do not want to sample only a subset of the population because
additional factors, such as racial demographics, are critical to the types of experiences students
have at NU. Utilizing a subset of the participant population would exclude information that
would essentially limit our data and any succeeding inferences that could be made.
Comparison Group
The scope of this evaluation does not include a comparison group. Only in its second year
and still a pilot program, Compass needs to be evaluated on whether or not it is accomplishing its
designed learning outcomes upon participants completion of the program. Once it is measured
whether participation in the program accomplishes the designed goals, then including a
comparison group would enhance the evaluation of the program, especially since it is believed
that first-year students who identify as low-income and/or first-generation who participate in
Compass feel a greater sense of belonging at NU than do their first-year, low-income and/or
first-generation peers who do not.
Research Design
We will be utilizing a longitudinal research design, one in which the same group of
participants will be analyzed over the course of one year (Banta & Palomba, 2015). A
longitudinal design was chosen because first-year low-income and/or first-generation students
have to apply to be a mentee in Compass and are expected to persist in the program through the
completion of their first year. This process will involve a pre and post-survey that participants
will be required to complete upon acceptance into Compass and upon completion of the program
at the end of their first year. Students will be required to complete the pre-survey after they have
received their acceptance into Compass to ensure we are analyzing the appropriate students who
will actually be a member of the program. This design will allow us to observe over time the
influence of participation in the program on achieving the programs short-term goals. Some of
the same assumptions that are to be made about the program also pose to be threats to our data in
this design. In particular, participants not persisting through to the completion of their first year
serves as the most prominent threat, so to help contest this, an intentional approach to the survey
design is taken to organize its distribution in a way that is simple for participants.
Survey Instrument and Implementation
Both the pre- and post-survey are constructed to provide measurable depictions of the
Compass programs learning outcomes. The survey matrix (Appendix D) provides a visual
representation of the relationship between each measurable item and learning outcome of the
program, which is then utilized in both surveys. The pre-survey (Appendix F) consists of 34
questions and should take approximately 10 to 15 minutes to complete. The post-survey
(Appendix H) consists of 46 questions and should take approximately 20 to 25 minutes to
complete. There is a combination of Likert scale questions and questions where participants are
asked to fill in the text box with their answers for both surveys. Questions were arranged into
sections based on which of the programs learning outcomes the questions were pertaining to in
order to easily guide participants through the surveys by maintaining visual order and
organization.
Survey Details
The purpose of the pre-survey is to gauge the self-reports of what the students who will
be participating in the program already know about campus resources and how they currently use
them, what organizations they are currently involved in, if any, and how they feel they are being
included or excluded in their experiences at NU. It is important to note that the Compass
program begins its operations three weeks after the start of the fall quarter, so it is appropriate to
assess the students accepted into Compass at this time. The post-survey serves to immediately
assess the same items as the pre-survey to see how completion of the program has influenced the
outcomes via self-reports. The data from both surveys will then allow us to conduct a statistical
analysis of the differences across surveys to determine how much change happened as a result of
participation in the program and essentially, of the effectiveness of Compass. The completion of
consent forms prior to participating in either survey will also be required (Appendices E and G).
Additionally, general questions that address other factors such as racial demographics are
included at the end of each survey to help make connections between and inferences about the
participants and what is self-reported. We include this information at the end of each survey
instead of the beginning so that these personally identifiable variables do not sway participants
answers based on what they think they should self-report.
Pilot Testing
For the sake of time, pilot testing could be risky in prolonging the ability to make
intentional improvements to the program in response to survey results. NU is set to increase the
14% of students who identify as low-income and/or first-generation to 20% by 2020, so it is
crucial to these students experiences that NU is ready to appropriately serve this population. We
could not pilot test these surveys on those who participated in Compass the past academic year
(2015-2016, its first year) because too much time has passed and the learning outcomes are
designed to be evaluated immediately upon completion of the program. Longitudinal designs are
stronger and more demanding so typically it is more feasible to use a design that is cross-
sectional or data collected on a population at a specific time (Banta & Palomba, 2015); however,
it would be most beneficial to the improvement of Compass to use a longitudinal design. The
comparison of the results between the pre and post-survey is critical in the assessment of the
influence of the program so administering one without the other would not be productive or
beneficial because it would not provide us with complete information.
To address these limitations, we would pilot test these surveys on a group of first-year
students who identify as low-income and/or first-generation who are not participating in the
program. We would identify two to three students who we will pilot the survey with to gain
feedback on item clarity, survey flow and overall experience. We would pilot this survey in
spring 2017 so that any necessary changes could be made during the summer of 2017. The actual
surveys will be administered in fall 2017 (pre-survey) and spring 2018 (post-survey). Program
revisions and improvements will be structured and addressed during the summer of 2018 to be
implemented in the 2018-2019 academic year, which would better prepare NU in time for the
20% by 2020 increase.
Survey Delivery
With assistance from NUs Student Affairs Information Technologys (SAIT)
partnership, the pre-survey will be administered online three weeks into the fall quarter at the
beginning of the Compass kick-off event in an on-campus computer lab to ensure that all
participants have access to a computer and to provide opportunities for students to ask evaluators
any questions that may come up. Likewise, the post-survey will be administered online at the end
of the academic year at the conclusion of the Compass closing ceremony in an on-campus
computer lab. Since this is a small population of students, providing participants with the space,
technology, time, and support to complete both surveys as part of the programs curriculum will
guarantee a 100% response rate. We will only incorporate one email reminder about the surveys
a couple of days in advance via email when communicating about that weeks Compass
workshop.
Director of SES Kourtney Cockrell in addition to Assistant Director Sharitza Rivera, who
directly manages the operations of the program and who will be listed on the surveys as the main
contact, will be present at both the Compass kick-off event and closing ceremony when the
surveys are distributed so they can explain the purpose of each survey, distribute the consent
forms (Appendices E and G), and answer any questions the participants may have. There will be
no financial or reward incentives for completion of either survey because the surveys will be
included as part of the Compass workshop curriculum. Additionally, when Kourtney Cockrell
and Sharitza Rivera share the purpose of the surveys, there will be an emphasis on how the
results of the surveys will inform future improvements of the program since Compass is still in
its pilot stage.
Statistical Analysis
Upon completion of gathering all of the data from both the pre- and post-survey, we will
analyze the results first utilizing descriptive statistics to calculate the mean averages and
frequencies within the data. We will describe our survey sample based on nominal variables;
ascribing numbers to data to assign them to groups of no particular order (Newcomer & Conger,
2010). For example, we will examine results across particular demographic groups like the
number of participants who identify with certain racial groups in addition to identifying as low-
income and/or first-generation to see the frequency and percentages of this data. This is
important to note because race absolutely influences student satisfaction and sense of belonging
(K. Cockrell, personal communication, October 3, 2016). Additionally, we will describe our
sample and data based on ordinal variables, variables that assume some ordered connection to
one another (Newcomer & Conger, 2010). For example, our surveys utilize some Likert scales so
we can examine the frequency and mean averages of which program participants can identify
appropriate campus resources. Also, to illustrate a relationship between both our nominal
variables and ordinal variables, we will examine, for example, how many black-identified
participants believe they feel an increased sense of belonging on campus.
Next, we will utilize inferential statistics to generalize to the entire population and to
investigate if there are statistically significant differences between the two different time points
of the pre- and post-survey. Specifically, we will use a paired-samples t test to determine this.
Additionally, we will also use ANOVA when we compare more than two mean scores to
determine whether these differ (Davidson, 2016). This will tell us whether participating in the
Compass program positively influenced the experiences of first-year low-income and/or first-
generation students. It can also tell us more about the low-income and/or first-generation
population at NU in general, which will be beneficial to preparing staff and faculty to support
this population as it continues to increase.
Presentation of Results
Since most of what we will be examining from the results are frequencies, percentages
and comparisons of means, we will use a combination of tables and charts to present our
findings. We will use tables and charts to represent our descriptive statistics of the number of
participants who identify with certain racial demographic groups, low-income and/or first-
generation, and how many participants feel similarly with particular items on the surveys. Also,
to connect the answers to the items on the Likert scales with demographics, we will use a cross-
tabulation to visualize these relationships. Moreover, we will use tables and charts to compare
the two mean scores across time to represent our inferential statistics.
Qualitative Approach
To supplement our quantitative approach, we will be conducting a qualitative analysis of
the effectiveness of the Compass program to measure whether involvement in the program
positively influences first-year low-income and/or first-generation students experiences. In this
qualitative analysis, participants will be asked to provide in-depth details about their experiences
in Compass and how their participation in the program has influenced the ways in which they
navigate the NU community. Data from our quantitative approach will provide us with statistical
representations of themes in students experiences, whereas the data from our qualitative
approach will illuminate elements that contributed to the statistical themes. To gather this
qualitative data, semi-structured interviews (SSIs) will be utilized, which includes a sequence of
open-ended questions where the interviewer is able to inquire further to encourage deeper
reflection of questions (Adams, 2010).
Though time consuming, SSIs will be used instead of focus groups because our
population of participants in Compass is small with 32 mentees. Additionally, SSIs will be more
inclusive of the diverse representation of mentees (i.e. race, gender, sexual orientation). If we
utilized a smaller sample size, it would be harder to make generalizations from the evaluation to
the larger group because we could have missed some identity representations in a smaller sample
size. Also, SSIs allows participants to feel more comfortable sharing deeper thoughts, feelings
and experiences with the interviewer because they will be sharing in a one-on-one setting where
confidentiality is more feasible than in a group setting. Participants may feel compelled to
answer in socially desirable ways in either a focus group setting or individual SSI setting;
however, given the nature of information we will be asking about in the interviews, SSIs are
more advantageous when asking probing questions about matters that our participants may not
be frank about if they were with their peers in a focus group (Adams, 2010). Instead of feeling
pressure to answer similarly to the group, participants will be able to give a more authentic
account of their personal experiences. To enhance our understanding of whether first-year low-
income and/or first-generation students feel a greater sense of belonging as a result of their
participation in Compass, we will administer our qualitative approach following the completion
of the program and quantitative surveys. This will allow us to better understand why our
participants responded in particular ways to the survey items.
Interview Protocol
To guide the SSI, an interview protocol (Appendix I) will be used that will allow us to
ask probing questions to encourage the participants to reflect at deeper levels. The interview
protocol was designed to address three program learning outcomes: identify and utilize campus
resources in times of need, describe at least one student organization, activity, program or
workshop in which they engaged with this year, and express greater satisfaction with their first
year experience at Northwestern than do peers who are not part of Compass (SES Compass,
n.d., p. 12). The design also addresses any assumptions and external factors that contributed to
the statistical data. Over the course of the sixty-minute interview, the interviewer will begin by
outlining the formalities of the interview. The interview script outlines the desired use of the
findings from the participants responses as well as their right to refrain from moving forward at
any time at their discretion. Once the participant has agreed to proceed, the interview begins with
a preliminary question followed by probing questions that will allow the participant to reflect on
their initial sentiments about attending NU. This approach is intentional with hopes that it will
uncover the attributes of participants ideal institutional structure, the external support they may
or may not have had as they began to finalize their decision, or any reservations they may have
held prior to making their commitment. Gathering this information will help inform whether the
Compass programs learning outcomes have been appropriately designed and whether the
program achieved its learning outcomes, thus addressing the expectations that students had
entering the institution and program.
The subsequent questions are designed to illuminate whether those expectations were met
or even exceeded through their involvement with the Compass program, in relation to the
programs learning outcomes. For each interview, the short-term goal that is directly aligned
with that segment of the interview is noted. Each of the questions is open-ended and focuses on
participants feelings about the influence of the Compass program. Time is allotted towards the
conclusion of the interview for the participant to share whether they believe the program
components aided in their successful transition and adjustment to the NU campus as desired. If
the participant believes that certain aspects of their first-year caught them by surprise due to a
lack of awareness, they will be provided a space to explain those experiences and what they
believe would have been helpful during the latter part of the interview in an effort to alleviate
that for future participants. Additionally, if there are any perceived areas of development they
feel would enhance or refine the influence of the program, there is an opportunity for them to
share that during this part of the interview. Finally, following the conclusion of the interview,
participants will be able to complete a demographic variable form (Appendix J). The form will
again capture, at a more intimate level, the characteristics of those who participated. Knowing
that the cohorts are comprised of a couple of students who hold very specific identities unlike
other participants, questions that would single out those particular students in a way that we
would be easily able to identify them from their surveys were omitted in the development of the
demographic form to proceed forward with confidentiality.
Pilot Testing
Before actual implementation of the protocol, we will complete a pilot test with a small
group (approximately two to three) of first-year students who identify as low-income and/or
first-generation, but who are not program participants. The pilot test would be implemented in
spring 2017. This will be important as it will help us gauge protocol item clarity, sequence, and
overall experience. Also, this will help us know whether our estimate for the interview duration
is accurate where it can be completed in an hour. To run through the interview protocol in its
entirety and to allow time for feedback, we will ask that students who participate in the pilot
testing be prepared to spend approximately one hour and fifteen minutes with us. Immediately
following the interview protocol, students will provide us with their feedback. To supplement the
pilot test results, we will consult with the Director of SES, Kourtney Cockrell, and the Assistant
Director of SES, Sharitza Rivera to identify whether the interview protocol addresses the
programs learning outcomes. Additionally, we will want to assess whether our interview
protocol will help us gain insights on how the intersectionality of different identities influences
participants experiences as they relate to the programs learning outcomes, highlight the quality
of some of the processes utilized to achieve our outcomes, and encourage more details about
what informed our participants responses to the quantitative surveys. Then, evaluators will
incorporate all of the information from the pilot test and SES staff to make any edits to the
interview protocol that are necessary in order to actually implement the protocol in fall 2017.
Sampling Frame
Similar to our quantitative approach, our qualitative approach will use a census in an
effort to get as much involvement from our program participants as possible. As mentioned
earlier, the Compass population is already small and we want to be inclusive of all who
participate in the program to better inform us of all of the experiences of this diverse population
(i.e. gender, sexual orientation and race in addition to their low-income and/or first-generation
identities). Though the goal is to interview all program participants, there is a possibility that not
everyone will persist through to program completion so we will want to select all of the
participants who have completed the quantitative pre- and post-surveys in the event that this is
the case, as the interview protocol is to supplement our quantitative data. Moreover, we will be
attentive to the saturation of similar reports being shared across multiple interviews to identify
themes, though this may be difficult to do given the number of participants in the program and
the diversity of identities. There is the possibility that we may not reach this saturation. This
limitation revisits one of the programs assumptions that those who participate in the program
share similar experiences.
Implementation Procedures
Since our pre- and post-surveys will be administered as part of the Compass curriculum
as shared in our quantitative approach, a brief introduction to our interview protocol will take
place during the Compass kick-off program following participants completion of the
quantitative pre-survey. Though the interview protocol will not take place until after the
completion of the program and completion of the post-survey, it is important to highlight the
programs novelty, the importance of gathering information from the participants about their
experiences so that the program can make any improvements and to foreshadow the timeline
when things will take place so participants can plan accordingly. A reminder email reiterating the
aforementioned information about the interview protocol will be sent out to all program
participants at the beginning of the spring quarter to, again, allow for participants to plan early.
Following the completion of the post-survey, which will be completed immediately following
the Compass program end of the year celebration, an e-mail invitation (Appendix K) will be sent
out to those participants who completed the post-survey, which should be all of the program
participants, to see who would be interested in participating further in our interview. The e-mail
invitation will also highlight free food and drinks as an incentive to participate. The SSIs will
take place two to three weeks following the completion of the program to allow time for
compiling data from the quantitative surveys, scheduling interviews, and allowing participants to
rest so that they are not experiencing survey fatigue during our interview protocol.
Director of SES, Kourtney Cockrell, and Assistant Director, Sharitza Rivera, will both
lead the interviews alternating between each in an effort to help eliminate some personal biases
that may be present in the delivery of the interview questions and responses to participants
questions. Both Kourtney and Sharitza identified as low-income and first-generation when
attending college so alternating interviewers will help alleviate some of their personal biases and
assumptions they may hold about what this experience means for the participants in the program
that could influence how they ask questions and how they reach certain themes. Though there
may be similarities, being low-income and/or first-generation at a private, elite institution
provides additional challenges that they both may not have experienced at their institutions when
they were attending. However, it is most ideal for Kourtney and Sharitza to moderate and lead
the interviews instead of outside evaluators because they will have built a certain level of rapport
and trust with Compass participants throughout the entire year by co-managing the program and
providing support for these students through all of SESs services in general. Granted, this could
pose a concern that students may not want to express their true thoughts; however, this
familiarity with Kourtney and Sharitza and the rapport built will allow participants to feel more
comfortable providing genuine and authentic answers during the interviews. It is also important
to note that participants will have already been familiar with providing on-going and honest
feedback about the program curriculum throughout the year following each cohort session as
they will have completed some form of post-assessment in regards to the workshops they
attended, so they will already feel a certain level of comfortability sharing their thoughts,
feelings and experiences. Additionally, the interviews will take place in the same buildings and
rooms that the weekly cohort sessions take place to continue to provide a welcoming, familiar,
and comfortable experience. The time of each interview will be dependent upon the participants
availability that will be scheduled as a result of participants responses to the e-mail invitation.
Though Kourtney and Sharitza will be alternating between interviews, both will be
present for each; however, one will serve as a moderator while the other will serve as the lead
who will be asking the questions. This will help to ensure that they also do not experience
interview fatigue as interviewers. While one person is leading, the other will help moderate, if
need be, but will also be taking notes. It is important to point out that the interviews will be
audiotaped; however, taking notes will help supplement the recordings in case something
happens to the recordings, but more significantly, to make any additional notes about
information shared that cannot be reflected in the recording. Despite participants being wary of
what may happen with the recording or thinking about the device during the interview,
audiotaping will allow the interviewee to be more actively engaged in the conversation (Adams,
2010), help the conversation feel more organic, and will save time as the lead interviewer will
not have to slow the conversation down just to take notes because the audiotape will be more
heavily relied on.
Data Analysis
To make meaning of the information collected through our qualitative approach, we will
need to analyze codes the process of examining qualitative data to label themes, concepts, and
categories of findings (Davidson, 2016). We will take a deductive approach, known as a priori,
to our coding where we will connect the development of initial codes to the assessment questions
we are asking (Davidson, 2016). As shared in the context of this evaluation plan, the Compass
programs framework is rooted in Dr. Mesmin Destins research pertaining to closing the social-
class achievement gap (K. Cockrell, personal communication, September 15, 2016). This
framework helped inform the programs learning outcomes, so it makes the most sense to utilize
a deductive approach beginning with major themes from the programs learning outcomes as
codes and address new themes as they emerge. This is known as descriptive coding, using the
programs outcomes as starting points for codes and building from there (Rogers & Goodrick,
2010). Prior to conducting the interviews, we will construct a map to develop our coding scheme
based on the programs learning outcomes as they relate to campus resources, campus
involvement, and inclusion (SES Compass, n.d.). This will help us categorize and view the
data in a way that allows us to identify patterns or problems within and across cases like those
pertaining to demographic characteristics, a method known as pattern coding (Rogers &
Goodrick, 2010). Following the completion of the interviews, we will give out demographic
forms to be completed to collect salient identity information that are relevant to how Compass
was experienced (i.e. race, gender, sexual orientation, school) that will help us match identifying
characteristics to our participants, which is known as attribute coding (Rogers & Goodrick,
2010).
In our approach, transcripts play a critical role in our ability to make meaning of our
qualitative data. We will analyze the data using a cutting and sorting technique where quotes that
seem important will be identified and cut from the transcripts (Rogers & Goodrick, 2010). We
will then write who said each quote, include contextual information on the back of the transcript
and then compile all of the transcripts coded similarly into groups. This technique is useful in
identifying sub-themes. It is important to incorporate inter-rater reliability in this process to
ensure consistency, which is the extent to which multiple coders coincide with one another how
they are coding (Davidson, 2016). Prior to beginning the full process of coding the transcripts,
the coders will each take copies of the same two to three transcripts, independently work on the
coding for those transcripts, and then come back together to discuss and make sure that
consistent themes are being identified by all. Moreover, following the completion of our actual
coding, we will employ member checking where we will send our initial interpretations of our
findings to see if they stimulate any further discussions (Rogers & Goodrick, 2010) to our
interview participants. Gaining their perspectives helps eliminate personal biases that may have
shaped how the information was analyzed and interpreted and is central to guaranteeing that we
remain focused on the goals of the evaluation.
As the SSIs are to be used to supplement our quantitative data, we will triangulate our
findings from our qualitative approach with the quantitative data collected as well as with
existing literature and research about this particular population. Triangulation is the process of
using multiple sources of evidence to create generalizations (Davidson, 2016). For example, let
us say that a student indicated on the post-survey, which was part of our quantitative approach,
that they believe they are appreciated by the campus community and then during the interview
shared that they made some close friends in the program. We could triangulate these data to
make the generalization that participation in the Compass program fostered a greater sense of
inclusion in the experiences of first-year low-income and/or first-generation students and connect
this generalization with existing literature and research to provide a theoretical lens as to why
this may be.
Limitations of Qualitative Approach
Possibly the largest limitation of our qualitative approach is interpreting the datas
meaning(s) appropriately and accurately. Given the diversity of the Compass population (i.e.
race, gender and sexual orientation in addition to being low-income and/or first-generation), it
can be complicated making accurate interpretations of the information that is shared in the
interviews by way of any personal biases, no matter how hard we work to try to eliminate this,
and participants may not have communicated sufficiently in their answers. Though we designed
the approach to use triangulation to help avoid this limitation, external factors, assumptions
and/or personal biases still play a major role in the how we make meaning of the data collected.
Also, time is a limitation in being able to conduct all of the interviews and be intentional in the
process. Depending on how the interviews go, regardless of pilot testing and how well it is
planned, there is still the possibility that all of the interview questions will not be addressed in
the time given.
Presentation of Results
To present our results, we will use a combination of tables and direct quotations. We will
use tables to make the results more aesthetically pleasing and easy to navigate. Tables will be
divided into three categories to represent the programs learning outcomes as they relate to
campus resources, campus involvement, and inclusion. Within each category, different sub-
themes will be highlighted with a direct quotation to support each. Additionally, we will present
findings from the SSIs in narrative form in our final assessment report, which would include
direct quotations, where we can explain our findings in depth. We would also be able to present
our examinations using pattern coding of any differences and similarities across groups.
Overall Limitations of Evaluation Plan
This evaluation plan offers SES an opportunity to examine a new hallmark program that
has not been assessed yet to measure whether Compass is achieving its learning outcomes and
whether first-year low-income and/or first-generation students who participate in the program
experience a greater sense of belonging at NU than do their first-year low-income and/or first-
generation peers who do not participate in the program. Though this is a great opportunity to
assess the effectiveness of the program, it is important to acknowledge that there are some
limitations that could hinder the value of the data. As mentioned throughout the plan, Compass is
a year-long program so there is the possibility that not all participants will persist through to the
completion of the program, which would negatively influence our census. Additionally, it will be
difficult to gauge whether participants accurately self-reported and to credit program
participants self-reported improvements in both the quantitative surveys and the qualitative
interviews to their participation in the program, rather than overall growth and maturation.
Another significant limitation is attributed to our outcomes-based approach. The processes
embedded in the program are not measured in relation to the outcomes; so upon completion of an
outcomes-based approach we only know whether outcomes were actualized and unable to
attribute whether these were actualized to any particular process. Furthermore, SSIs provide a
limitation in that they are time consuming and require interviewer sophistication where the
interviewers are sensitive and knowledgeable (Adams, 2010).
Despite the limitations to our evaluation plan, we have designed our approaches in such a
way that will help alleviate these. We will be surveying our participants at multiple points and
triangulating our quantitative data with our qualitative data as well as with existing literature and
research to strengthen our claims. We designed our approaches in a way that would be less of a
hassle for our participants to be surveyed by incorporating our pre- and post-survey into the
curriculum of Compass so that participants are not required to give additional time. For our SSI,
we will be purchasing food and drinks for participants for spending additional time outside of the
curriculum to participate which will help ensure full participation from our census. We were
intentional to set up resources and cultivate certain spaces and relationships that are welcoming
to participants so that they feel comfortable self-reporting. Additionally, in regards to the
limitations of our outcomes-based approach, we can still obtain helpful data that informs our
efforts. Since this is the first formal assessment of this pilot program, it makes the most sense to
begin with an outcomes-based approach and depending on the results, there can be subsequent
inquiries about the processes that impact these outcomes later. Moving forward, we would hope
to incorporate a comparison group in our assessment that would inform us whether first-year
low-income and/or first-generation students who participate in Compass feel a greater sense of
belonging at NU than do their first-year low-income and/or first-generation peers who do not.
Timeline
While the proposed timeline has been indicated throughout our evaluation plan, a more
formal and visual representation of this assessment timeline from now through the end of the
2018-2019 academic year can be found in Appendix L. In general, our pilot testing for our
quantitative surveys and our SSI protocol will take place in the spring quarter of 2017. Following
our gathering of feedback, any necessary changes will be made in the summer of 2017. Then, we
will administer our final quantitative pre-survey in fall 2017, our final post-survey in spring
2018, followed by conducting our final SSI protocol three weeks later in spring 2018. Data will
be gathered and analyzed summer 2018 so that program improvements can be made and
implemented for the 2018-2019 academic year.
Budget
Materials applicable to the assessment of the program would consist of the development
of the pre-survey, post-survey, the respective consent forms and the SSI materials, all of which
will be handled by the SES staff. The work being completed by the department staff will reduce
the overall cost of the program assessment. Once the content is developed, colleagues from NUs
Student Affairs Assessment department will review the documents and assist with transcription
and coding, which allows the administrators of the assessment to again consolidate funds by
leveraging institutional resources.
While consideration has been made to ensure that not all aspects of the assessment would
require funding, there are certain components over the course of the year that are recommended
to receive funding. As an incentive for participating in the SSIs outside of the Compass
curriculum, food and drinks will be provided at $12 per person. Also, upon time to conduct the
SSIs, to ensure accuracy for transcription, five recorder devices and supplemental materials will
be requested which would total approximately $200. The task of transcription will be conducted
by the SES staff with the help of NU Student Affairs Assessment department, again fraying cost.
A complete overview of the assessment expenses can be found in Appendix M, keeping the
overall expenses of program assessment to $584.

Next Steps
Upon completion of the assessment, the data would be gathered and coded to better
inform the Compass program, SES staff, and NU community of future implications to program
development. During the summer following data collection, the curriculum, policies and
procedures will be reviewed for continuity and refined as needed. For example, the language
used for the programs learning outcomes should be refined to omit any ambiguity. Also, in the
interest of continuous improvement, the assessment cycle would continue for the following year.
Components such as the number of participants and diversification of the demographics of the
cohorts may fluctuate as the program progresses. Similarly, as the incoming freshmen population
increases, the scalability of the program will also be evaluated to further support the needs of the
Compass programs target population.

References
Adams, W.C. (2010). Conducting semi-structured interviews. In J.S. Wholey, H.P. Hatry, &
K.E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 365-377). San
Francisco, CA: Jossey-Bass.
Banta, T.W., & Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and
improving assessment in higher education (2nd ed.) San Francisco, CA: Jossey-Bass.
Campus Inclusion & Community, & Campus Partner Planning Group (2016). Strategic Plan
2016-2022.
Davidson, L. (2016). Qualitative data analysis [PowerPoint slides]. Retrieved from
https://sakai.luc.edu/portal/site/ELPS_431_001_2043_1166/page/fbf79438-8d48-4550-
9865-0eed77cd6e08
Davidson, L. (2016). Using descriptive + inferential statistics in assessment [PowerPoint slides].
Retrieved from
https://sakai.luc.edu/portal/site/ELPS_431_001_2043_1166/page/fb1197a5-6746-48f1-
845e-01ef56aa6403
Kellogg, W.K. (2004). Logic model development guide. Battle Creek, MI: W.K. Kellogg
Foundation.
Newcomer, K.E. & Conger, D. (2010). Using statistics in evaluation. In J.S. Wholey, H.P. Hatry,
& K.E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 454-492). San
Newcomer, K.E. & Triplett, T. (2010). Using surveys. In J.S. Wholey, H.P. Hatry, & K.E.
Newcomer (Eds.), Handbook of practical program evaluation (pp. 262-297). San

Northwestern University. SES Compass Program: CURRICULUM. n.d.
Rogers, P.J., & Goodrick, D. (2010). Qualitative data analysis. In J.S. Wholey, H.P. Hatry, &
K.E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 429-453). San
Stephens, N.M., Hamedani, M.G., & Destin, M. (2014). Closing the social-class achievement
gap a difference-education intervention improves first-generation students academic
performance and all students college transition. Psychological science, 25(4), 1-18.
Student Affairs Assessment, & Campus Inclusion and Community. (2013). Undergraduate
students from low-income family backgrounds and the Northwestern experience.
Tinto, V. (1997). Classrooms as communities: Exploring the educational character of student
persistence. The Journal of Higher Education. 68(6), 599-623.

Appendix A- Division of Student Engagement Organizational Chart

Appendix B- Logic Model
OUTPUTS OUTCOMES + IMPACT

SITUATION INPUTS Short-term (Learning/ Medium-term Long-term
Activities Participation
Awareness) (Action) (Conditions)
What is the issue or What we Activities What we Participation What are the short- What are the What is the
need driving the invest (i.e. do (Appendix C): Who we reach: term results? medium-term ultimate impact?
evaluation? people, Mandatory weekly Immediate upon results? Attainable within
knowledge, cohort meetings with 32 first-year program completion: Attainable within 4-6 years:
Mentors gained more resources, all mentors, mentees low-income Identify and utilize 2-3 years: Identify ways in
than mentees so the materials): and staff of each and/or first- campus resources in Mentees seek which mentees can
evaluation will focus cohort where mentors generation times of need; describe leadership stay connected to
on the attainment of Staff (SES), students as at least one student opportunities
and SES staff will Compass/resources/
the learning outcomes campus mentees (2 organization, activity, within the
facilitate workshops staff even after
as achieved by the partners, cohorts of 16); 9 program or workshop program or in
that pertain to their first year;
mentees and their funds, time upperclassmen in which they engaged other avenues at
community building, overall increased
overall satisfaction (year-long low-income with this year; express NU following
campus resources, and knowledge and
with their NU program), and/or first- greater satisfaction their first-year
involvement; increase in
experience resources generation with their first year experience in the
Mandatory weekly personal,
Who is affected by (online portal students as experience at program;
coaching sessions for professional and
the issue? Mentees of all campus mentors (4 or 5 Northwestern than do increased
the mentors where they academic network
Who is interested in resources for per cohort); peers who are not part knowledge and
receive continued (social and cultural
the issue (who are the example), There is an of Compass. For the increase in
training from SES staff capital) that helps
stakeholders)? stories about application third learning personal,
on topics such as them positively
Donors, campus lived process for outcome, we propose professional and
facilitation and navigate the NU
partners (i.e. residence experiences mentors and new language instead academic
mentorship; community and
life, financial aid), from mentees and that is more network (social
Mandatory biweekly culture
VPSA, AVP, CIC, upperclassme from the measurable: articulate and cultural
n who one-on-ones that the
SES, mentors, mentors must hold application a greater sense of capital) that helps
mentees, low-income identify as pools, the inclusion within the them positively
low-income with their mentees for
and/or first-generation an hour at minimum; Director and campus community navigate the NU
students as a whole and/or first- Assistant throughout their first community and
generation, Mandatory weekly
weekend events Director choose year experience at NU culture
continued the 9 mentors than do their peers
training facilitated by mentors
yet optional for and 32 mentees who are not part of
mentees to attend. Compass.
Assumptions: (Beliefs, expectations, and principles that guide our work)
1. All of the participants share similar experiences as low-income and/or first-generation students as it pertains to navigating the Northwestern
experience
2. All of the participants (mentors and mentees) will persist through to the completion of the program (full academic year)
3. Mentor-mentee matches are sufficient for the development of the mentees and their needs
4. Delivery of the program content is facilitated in a way that is conducive to all learning styles and is received well by all participants
5. Participants can accurately self-report their experiences and satisfaction as it relates to their involvement in the program
Environment: (Influential factors, either supportive or antagonistic)
1. Intersectionality of multiple identities (i.e. non-first-generation American, race, gender, etc.)
2. Academic rigor of the prestigious institution and its impact on physical and mental health of participants
3. Additional involvement on campus
Adapted from The University of Arizona, College of Agriculture & Life Sciences:
https://extension.arizona.edu/evaluation/content/logic-model-worksheets)
Appendix C Compass Program Curriculum Snapshot
Compass Program Snapshot

Fall Mentor Sign- up Workshop Topics
Kick-Off Retreat: Community building; Celebrating You; SES Intro and
Logistics
Workshop #1: Mentor-Mentee Matches
Workshop #2: Strengths of First Gen/Low-Income Students
Workshop #3: Avoiding Midterm Stress
Workshop #4: Mental Health Check-In
Workshop #5: Football Tailgate (Family Weekend)
Workshop #6: Who are we as COMPASS
Workshop #7: Faculty Mixer
Workshop #8: Bowling Party
Winter Mentor Sign- up Workshop Topics
Workshop #1: Winter Kick-Off/Regroup
Workshop #2: Mental Health Check-In
Workshop #3: Intro to SE
Workshop #4: Financial Aid - Moving Off Campus & Budgeting 101
Workshop #5: Study Abroad

Workshop #6: NU Mens Basketball Game and Tailgate
Workshop #8: Chicago Outing
Workshop #9: Wrap-up Celebration and Reflection
Spring Mentor Sign- up Workshop Topics
Workshop #1: Spring Kick-Off
Workshop #2: SOLO- Dinner
Workshop #3: Civic Engagement/LCE
Workshop #5: Wrap Up/ End of Year Retreat

Appendix D Survey Construct Map and Matrix

Student Enrichment Services
Survey Construct Map & Item Matrix
Answer
Logic Model Component CM Q# Question Answer Options
Type
Short Term Goals
I feel that I am capable of recognizing when I need
additional assistance regarding__________matters. Strongly Disagree
Academic Disagree
1-4 Scale
Personal Neutral
Financial Agree
Professional Strongly Agree
I feel that I am comfortable with seeking assistance
regaridng ____________ matters. Strongly Disagree
Academic Disagree
5-8 Scale
Personal Neutral
Financial Agree
ST-
For __________ support, I am confident in my ability to
Identify and utilize campus resources in
identify the approrpaite campus resource. Strongly Disagree
times of need
Academic Disagree
9-12 Scale
Personal Neutral
Financial Agree
Never
How frequently do you utlize _______ resources? Less than 1 time per week
Academic 1-2 times per week
Personal 3-4 times per week
13-17 Scale & Text Box
Financial 5-6 times per week
Support
Professional Please list any specific associated
services.
0
1
How many extra-curricular organizations do you
ST- 18 Multiple Choice 2
participate in?
Describe at least one student 3
organization, activity, program or 4+
workshop in which they engaged with Please describe the nature of one organization,
19 program or workshop you have participated in, if you Text Box
have attended one at NU since starting here.
Strongly Disagree
Disagree
20 I feel a sense of belonging on campus Scale Neutral
Agree
Strongly Agree
Peers
I value having trusting relationships with
my____________ Strongly Disagree
21-23 I value feeling supported by my _________ Scale Disagree
I value having relatioship with my ________ that Neutral
highlght our similarities. Agree
ST- Strongly Agree
express greater satisfaction with their NU Staff
first year expereince at NU than do peers I value having trusting relationships with the
who are not part of Compass ____________ Strongly Disagree
24-27 I value feeling supported by the _________ Scale Disagree
I value having relatioship with the ________ that Neutral
Strongly Agree
NU Faculty
I value having trusting relationships with the
____________ Strongly Disagree
28-30 I value feeling supported by the _________ Text Box Disagree
I value having relatioship with the ________ that Neutral
Strongly Agree
31 What is your University NetID ? Text Box
32 Do you identify as low-income? * Check Box Yes/ No/ I preferr not to answer
33 Do you identify as first-generation? * Check Box Yes/ No/ I preferr not to answer
American India
Asian
General Information Black or African American
Multiple Pacific Islander
34
Choice/ Text Box White
I Prefer Not To Answer
Text Box: 'If not specified above,
Please specify your race(s) Select all that apply: please detail below:'
Appendix E Pre-Survey Consent Form
COMPASS PROGRAM CONSENT FORM

PRE-SURVEY
You are invited to participate in a confidential survey interested in your experience thus
far as a first-year student at Northwestern University and upcoming participation in the Compass
program.
The survey will ask you questions regarding your current experiences at Northwestern.
You will be asked about your current involvement and use of campus resources.
You are encouraged to complete the survey within one sitting. If you require additional
time, we encourage you to speak with the staff present while taking the survey. The survey may
take anywhere between 10 to 15 minutes to complete.
If you volunteer to participate in the study, it is your right to withdraw and stop
participating at any time you wish. You will not be penalized for choosing to do so.
All information that you provide will be kept confidential and solely used to improve the
program here after. There will be no personally identifiable information shared with anyone
other than the research team.
If you have any questions regarding the study, or regarding your rights as a participant,
your main point of contact is Assistant Director Sharitza Rivera at
sharitza.rivera@northwestern.edu
AGREEMENT TO PARTICIPATE IN RESEARCH
______________________ _____________
Participants Name Date
______________________ _____________
Signature of Participant Date
______________________ _____________
Signature of Facilitator Date
Appendix F Pre-Survey
COMPASS PROGRAM
PRE-SURVEY
The Compass program is a year-long peer mentor program geared towards supporting the
transition and integration of first-year low-income and/or first generation students onto the
Northwestern University campus. The purpose of this survey is understand your initial thoughts
as it pertains to your experience here at the university. Your response will assist in the
integration of campus resources and overall program improvement.
Campus Resources
Ability to identify and utilize campus resources
Please indicate the degree to which you agree with the Strongly Strongly
Disagree Neutral Agree
following statements by checking the box that applies. Disagree Agree
I feel that I am capable of recognizing when I

need additional assistance regarding
academic matters.
need additional assistance regarding personal
matters.
need additional assistance regarding financial
matters.
need additional assistance regarding
professional matters.
I feel that I am comfortable with seeking
assistance regarding academic matters.
assistance regarding personal matters.
assistance regarding financial matters.
assistance regarding professional matters.
For academic support, I am confident in my
ability to identify the appropriate campus
resource.
For personal support, I am confident in my
resource.
For financial support, I am confident in my

resource.
For professional support, I am confident in my
resource.
Less than 1
1-2 times 3-4 times 5-6 times
Never time per
per week per week per week
week
How frequently do you utilize academic

resources?
Please list any specific associated services.
How frequently do you utilize personal

resources?
How frequently do you utilize financial

resources?
How frequently do you utilize support

services?
How frequently do you utilize professional

services?
Extra-Curricular Involvement
*Extra-curricular: Participation in additional activities outside of regular course work
0 1 2 3 4+
How many extra-curricular organizations do

you participate in?
Please describe the nature of one
organization, program or workshop you have
participated in, if you have attended one at
NU since starting here.
Satisfaction
Sense of fulfillment or pleasure with ones experience
I feel a sense of belonging on campus.

I value having trusting relationships with my
peers.
I value feeling supported by my peers.
I value having relationships with my peers
that highlight our similarities.
NU staff.
I value feeling supported by the NU staff.
I value having relationships with the NU staff
that highlights our similarities.
NU faculty.
I value feeling supported by the NU faculty.
I value having relationships with the NU
faculty staff that highlights our similarities.
General Information
What is your University NetID?
Do you identify as low-income?
(Low-income is defined as a family of four

with one in college making $40,000 per year
or less, but its a sliding scale based on the

number in a family and other wealth-related
factors beyond income (Student Affairs
Assessment & Campus Inclusion and
Community, 2013, p.3).)
Do you identify as first-generation?
(First-generation is defined as a college

students who do not have parents with 4-year
college degrees (Stephens, Hamedani &
Destin, 2014).)
Please specify your race(s):

Choose an item.
Select all that apply:
Appendix G Post-Survey Consent Form
COMPASS MENTOR PROGRAM CONSENT FORM

POST-SURVEY
You are invited to participate in a confidential survey interested in your experience as a

first-year student at Northwestern University and participation in the Compass program.
The survey will ask you questions regarding your experiences over the course of your
first year. You will be asked to quantify your involvement and use of campus resources.
Additionally, you will be asked to rate your overall satisfaction with the Compass mentor
program and its components.
You are encouraged to complete the survey within one sitting. If you require additional
time, we encourage you to speak with the facilitator. The survey may take anywhere between 20
to 25 minutes to complete.
If you volunteer to participate in the study, it is your right to withdraw and stop
participating at any time you wish. You will not be penalized for choosing to do so.
All information that you provide will be kept confidential and solely used to improve the
program here after. There will be no personally identifiable information shared with anyone
other than the research team.
If you have any questions regarding the study, or regarding your rights as a participant,
your main point of contact is Assistant Director Sharitza Rivera at
sharitza.rivera@northwestern.edu
AGREEMENT TO PARTICIPATE IN RESEARCH
______________________ _____________
Participants Name Date
______________________ _____________
Signature of Participant Date
______________________ _____________
Signature of Facilitator Date
Appendix H Post-Survey
COMPASS PROGRAM
POST-SURVEY
Campus Resources
Ability to identify and utilize campus resources
NUs Compass program increased my ability

to recognize when I need additional
assistance regarding academic matters.
assistance regarding personal matters.
assistance regarding financial matters.
assistance regarding professional matters.
NUs Compass program increased my comfort
with seeking assistance regarding academic
matters.
with seeking assistance regarding personal
matters.
with seeking assistance regarding financial
matters.
with seeking assistance regarding professional
matters.
For academic support, I am confident in my
resource.
For personal support, I am confident in my
resource.
For financial support, I am confident in my

resource.
For professional support, I am confident in my
resource.
Less than 1
Fall Quarter Never time per
week
How frequently during the fall quarter did you

utilize academic resources?

utilize personal resources?

utilize financial resources?

utilize support services?

utilize professional services?

Less than 1
Winter Quarter Never time per
week
How frequently during the winter quarter did

you utilize academic resources?

you utilize personal resources?


you utilize financial resources?

you utilize support services?

you utilize professional services?

Less than 1
Spring Quarter Never time per
week
How frequently during the spring quarter did

you utilize academic resources?

you utilize personal resources?

you utilize financial resources?

you utilize support services?

you utilize professional services?
Extra-Curricular Involvement
*Extra-curricular: Participation in additional activities outside of regular course work
0 1 2 3 4+
How many extra-curricular organizations do

you participate in?
Please describe the nature of any

organizations, programs or workshops you
have participated in.
Satisfaction
Sense of fulfillment or pleasure with ones experience
Participating in NUs Compass program

increased my sense of belonging on campus.
Participating in NUs Compass program helped
me develop trusting relationships with my
peers.
allowed for me to feel supported by my peers.
provided a space to engage in relationships
with my peers that highlight our similarities.
me develop trusting relationships with the
Compass program staff.
allowed for me to feel supported by the
Compass program staff.
with the Compass program staff that
highlights our similarities.
me develop trusting relationships with the NU
faculty.
allowed for me to feel supported by the NU
faculty.
with the NU faculty staff that highlights our
similarities.
Overall
General feedback regarding the Compass program
The range of services highlighted during the
Compass program met my expectations.
The quality of services highlighted during the
Compass program met my needs.
I believe that the exposure to services through
the Compass program aided in my ability to
be successful during my first year.
If you have any additional comments or
suggestion that could enhance the services
provided by the NU Compass program, please
detail them here.
General Information
What is your University NetID?
Do you identify as low-income?
(Low-income is defined as a family of four

with one in college making $40,000 per year
or less, but its a sliding scale based on the

number in a family and other wealth-related
factors beyond income (Student Affairs
Assessment & Campus Inclusion and
Community, 2013, p.3).)
Do you identify as first-generation?
(First-generation is defined as a college

students who do not have parents with 4-year
college degrees (Stephens, Hamedani &
Destin, 2014).)
Please specify your race(s):

Choose an item.
Select all that apply :
Appendix I SSI Interview Protocol
One-on-one Semi-Structured Interview (SSI) Protocol
Focus of the Interview: To deeply examine the impact of the Student Enrichment Services
(SES) Compass program on the successful navigation of the Northwestern University
(NU) campus by first-year low-income and/or first-generation students.
Specific Foci of the Interview:
1. Are participants able to identify what constitutes time of need for them emotionally,
socially, and/or academically.
2. Illuminate any limitations perceived by the participants as it pertains to their ability to
become connected with campus clubs or organizations
3. Students sense of inclusion within the campus community throughout their first year at
NU as it pertains to their affiliation with the Compass program.
Interview Script
Thank you for your willingness to participate in our interview today. My name is _______
(insert either Kourtney Cockrell or Sharitza Rivera, whomever is taking the lead) and I will serve
as the facilitator for this interview. You may be familiar with me from my role as _______
(insert Director or Assistant Director of SES, whomever is taking the lead) and in my
supervision of the Compass peer mentoring program this year. Over the course of this year, the
Compass program has been geared towards aiding in your integration to the Northwestern
University campus, understanding and attainment of campus resources and overall enhancement
of your first-year experience through peer and staff interactions. During our time today, we hope
to add more context to the feedback we received overall from you and your peers in our initial
survey. All of your contributions thus far has been truly invaluable and after the conclusion of
today we are eager to find ways to improve the program in terms to come.
The interview should take approximately 60 minutes. If at any time you wish to end this
interview or skip any question, you are surely at liberty to do so. We do anticipate sharing our
findings with the staff of Student Enrichment Services as a foundation for improvement.
Considering such intentions, we want to encourage you to be open and honest in your responses.
While we will share our overall findings, your individual sentiments and identity will remain
confidential. As I conclude our interview overview, I ask that you review our interview consent
form to ensure you understand the details of our time together. By signing the form you consent
to participate in this interview.
Nature of the interview
Thank you for completing the interview consent form. To provide you with further clarity, I
would like to give you an overview of the interview structure. I will ask you a series of questions
aimed to better inform our understanding of the program structure, perceived benefits and
learned outcomes of the participants. As you share your experiences, I may inquire for further
clarification or expand on your remarks with a follow up question to provide a deeper

understanding. Again, if at any time you wish to discontinue speaking about a certain point due
to discomfort, please do not hesitate to share that with me, at which time we can move forward
with our preexisting questions.
Introductory Question
AS WE BEGIN, CAN YOU BRIEFLY SHARE SOME OF THE CONTRIBUTING FACTORS THAT INFLUENCED
YOUR DECISION TO COME TO NORTHWESTERN UNIVERSITY?
Intentional mention of their confidence in institution commitment, if need be, the following
probes could allow for the participant to elaborate further:
What were some characteristics you sought in the institution of your choice?
How supportive did your family seem in your decision?
Were there any reservations you had regarding your commitment to NU?
What factors did you weigh in making your final decisions?
Can you describe, if any, what type of connection did you have to NU prior to attending?
Focus 1: As participants able to identify what a time of need may be for them emotionally,
socially, and academically.
Short Term Goal: Identify and utilize campus resources in times of need
I. During your first year at NU, how did you become aware of campus resources geared
towards your academic success? Personal success? Financial success? Probe: How do
you believe the Compass peer mentoring program assisted in this awareness?
A. Over the course of this past year, could you describe a time where you felt in need of
emotional/social support?
B. Over the course of this past year, how would you define a time of need for academic
support?
C. Can you describe a time(s) when you noticed you needed emotional support versus
academic support? Probe: Which were you able to seek assistance with more often?
Probe: Why do you believe that is?
D. In what ways do you feel that, outside of your involvement with the Compass
program, you have been able to make a significant connection with campus resources?
II. Were any of the components of the Compass program (workshops, one-on-one
relationships, and activities) helpful? If so, which components? Probe: Did you feel that
any of these components directly supported your understanding of campus resources?
Focus 2: Illuminate any limitations perceived by the participants as it pertains to their ability to
become connected with campus clubs or organizations.
Short Term Goal: Describe at least one student organization, activity, program or workshop in
which they engaged with over the course of their first year.
I. What attributes did you hear of or observe within the club or organization that intrigued
your participation?
II. What do you believe aided in your interest to get involved in extra-curricular
programming/clubs/organizations?
A. Can you share about when, time wise, throughout the semester you began to seek
campus involvement?
III. Were there any barriers to your ability to become involved around campus? If so, please
share.
Focus 3: Students sense of inclusion within the campus community throughout their first-year
at NU as it pertains to their affiliation with the Compass program
Short Term Goal: Express greater satisfaction with their first-year experience at NU than their
peers who did not participate in Compass program
I. In what ways do you feel that through your participation in the Compass program your
first-year experience at NU was impacted?
A. What do you believe makes for a satisfying first-year experience?
B. How would you define satisfaction with either a service or program?
C. What characteristics do you believe make a relationship significant to you? Probe: Is
there a difference among relationships you have built with peers versus faculty or
other campus representatives? Probe: If so, how do you believe they vary? Probe:
Which relationships have the most relevance to you at this time?
II. If you were to converse with your peers who were also in their first year that did not
participate in the Compass program, in what ways do you believe their first-year
experience would be similar or different from yours as a participant in the Compass
program?
III. What aspects of the Compass program were the most helpful? Probe: for what reasons?
IV. What aspects of the Compass program do you believe were the least impactful? Probe:
for what reasons?
V. What modifications to the Compass program do you believe would enhance future
experiences within the program?
Conclusion
That concludes our interview questions. From what you have shared, it is evident that your first
year has provided quite a bit of experiences that may have an impact on how you enter into your
second year. Aside from this setting, have you had an opportunity to engage in a debriefing
dialogue, be it formal or informal, regarding your experiences?
Is there any additional points you wish to share that perhaps were not covered earlier in our
conversation?
I, on behalf of NU, SES, the Compass program, and our research team, would like to say thank
you for participating today in our interview. We truly could not improve as we move forward
without the input of students like yourself. We wish you the best with the remainder of the term
and going into the summer. If any additional questions arise regarding this process, we
encourage you to contact us.
Appendix J Demographic Variable Form
NORTHWESTERN UNIVERSITY, STUDENT ENRICHMENT SERVICES

COMPASS PEER MENTOR PROGRAM PARTICIPANT DEMOGRAPHIC FORM
WHAT IS YOUR
RACIAL
SEXUAL NU
IDENTIFICATION SEX^ GENDER^^
ORIENTATION COLLEGE/SCHOOL
(SELECT ALL THAT Choose an item. Choose an item.
Choose an item. Choose an item.
APPLY)
CHOOSE AN ITEM.
IF NOT LISTED IF NOT LISTED IF NOT LISTED IF NOT LISTED IF NOT LISTED
PLEASE INDICATE PLEASE INDICATE PLEASE INDICATE PLEASE INDICATE PLEASE INDICATE
BELOW BELOW BELOW BELOW BELOW
CLICK HERE TO ENTER CLICK HERE TO ENTER CLICK HERE TO ENTER CLICK HERE TO ENTER CLICK HERE TO ENTER
TEXT. TEXT. TEXT. TEXT. TEXT.
LOW-INCOME
FIRST GENERATION A family of four with one in college making $40,000
College students who do not have parents with 4- per year or less, but its a sliding scale based on the
year college degrees (Stephens, Hamedani & number in a family and other wealth-related factors
Destin, 2014). beyond income (Student Affairs Assessment &
Campus Inclusion and Community, 2013, p.3).
Choose an item.
Choose an item.
^Sex: References an individuals reproductive system

^^Gender: References an individuals social role or personal identification based on an internal
implications
*Transwoman: An individual who is assigned male at birth, but has a feminine gender identity.
** Transman: An individual who is assigned female at birth, but has a masculine gender
identity.
***Genderqueer: a person whose gender identity is neither man nor woman, is between or
beyond genders, or is some combination of genders. Often includes a political agenda to
challenge gender stereotypes and the gender binary system.
(Adapted from American University, The Center for Diversity & Inclusion, Office of Campus Life:
http://www.american.edu/loader.cfm?csModule=security/getfile&pageid=4045419)
Appendix K SSI E-mail Invitation
Dear _________ (program participant),
You are invited to participate in an interview to allow us to learn more in-depth about
your participation in the Compass program.
The interview is designed to ask you questions about your experiences in the program
and at Northwestern University, at large, during your first year. You will be asked to provide
information about your use of campus resources, any campus involvements and your sense of
belonging to the campus community.
The interview will take place any time between _______ (insert date) and ______ (insert
date) with what works best for your availability. The interview will take approximately 60
minutes and will be administered by Kourtney Cockrell and Sharitza Rivera.
As we recognize that we are asking for additional effort and time on your part, we are
providing free food and drinks.
All information that you provide will be kept confidential and solely used to make
improvements to the program for the future. If you volunteer to participate in the interview, it is
your right to withdraw and stop participation at any time you wish. You will not be penalized for
choosing to do so.
If you are looking to participate, please respond to this email no later than ________
(insert date) by _______ (insert time) of your interest as well as your availability during the time
frame mentioned above. More information will be shared with you upon confirmation of your
participation and scheduling of your interview appointment.
If you have any questions regarding the interview, or regarding your rights as a
participant, your main point of contact is Assistant Director, Sharitza Rivera at
sharitza.rivera@northwestern.edu.
Best,
Student Enrichment Services

Appendix L Assessment Timeline
2017 2017 2017 2017 2017 2017 2018 2018 2018 2018
ACTIVITY Jan. Feb. April May June Oct. June July Aug. Sept.
Review evaluation plan X
with SES
Review surveys and SSI X X

protocol with SES
Pilot test surveys and SSI X
protocol
Revise surveys and SSI X
protocol based on
feedback
Finalize evaluation plan X
Administer pre-survey at X
Compass kick-off
Analyze pre-survey results X
Administer post-survey at X
Compass end of year
celebration
Analyze post-survey X
results and compare with
pre-survey
Email invitations for SSIs X
Schedule SSIs and follow X
up emails to non-
responders
Conduct SSIs X X
Transcribe interviews X
Code transcripts X X
Member check X
Final analysis of data X
Determine X
recommendations for
Compass
Present recommendations X
Appendix M- Budget
Item Quantity Total Cost

Incentives for SSI $12/ person (32 mentees total) $384
(Food/Beverages)
Pre-Survey Materials $0 (Services provided by SES
staff)
Post-Survey Materials $0 (Services provided by SES
staff)
SSI Materials $0 (Services provided by SES
staff)
Transcription $0 (Services provided by SES
staff)
Recording 5 $200
Devices/Materials
Total $584
Appendix N- Presentation

Compass Evaluation Plan - Ellington

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Compass Evaluation Plan - Ellington

Uploaded by

Copyright:

Available Formats

Running head: AN EVALUATION PLAN FOR COMPASS

Loyola University ChicagoTrebby Ellington & Alyssa D. Humbles

Statement of Problem and Significance

Northwestern University (NU), a selective, private, predominantly white research

addition to its academic rigor. In particular, students of underrepresented and traditionally

marginalized populations (i.e. black, LGBTQIA, low-income) encounter dissatisfaction at NU at

demonstrate an increased sense of belonging to the Northwestern community and its

surrounding neighborhoods, develop skills to utilize campus resources to navigate communities,

campus and key components of the program.

Vice President of Student Affairs Patricia Telles-Irvin, Ed.D. was recruited to NU in

the university community to foster inclusive learning environments through intentional

Engagement (Appendix A).

office and its services.

Through campus-wide partnerships, SES builds an inclusive Northwestern community by

support, fellowships and research, involvement, Northwestern Career Advancement, school-

Compass Program Details

and/or low-income backgrounds. Through one-on-one meetings, social events, and

workshops, students learn how to successfully navigate the Northwestern experience

beyond. (p. 12).

communication, September 15, 2016).

and/or low-income accompanied by a four-year warranty (K. Cockrell, personal communication,

September 24, 2016).

background specific. Utilizing a difference-education intervention, they used a panel of senior

wide partnerships. To assist low-income and/or first-generation students in navigating the NU

inclusive to all students at NU.

Compass Assessment Rationale

outcomes at the conclusion of the program.

Logic Model Description

participation in the program as the mentors.

their responsibilities as a student, they self-elect to also add supplementary leadership

of this caliber to fruition.

place so that the following outcomes can be addressed in a deliberate fashion.

who are not part of Compass.

undergraduate trajectory at NU beyond their first-year mentee experience. Furthermore, the

first-generation students as it pertains to navigating the Northwestern experience. Next, we

their experiences as it relates to their involvement with the program.

Considering a key component of the program is contingent on being low-income and/or

first-generation, the intersectionality of multiple identities (i.e. non-first-generation American,

important to disregard. To coincide with the mentees participation in the program, we

attainment of the same objectives outlined by this program.

created to achieve. An outcomes-based assessment suggests gathering specific measures to

Compass is achieving its learning outcomes.

Furthermore, it is important to ask: does the new conceptual framework of integrated

at the conclusion of the program to track incremental shifts.

Reasoning for Approach

first-generation students to apply to be a mentee in the program; at which point minimal

who identify as low-income and/or first-generation.

low-income and/or first-generation feel a greater sense of belonging at NU by participating in the

and statistical analysis.

first-generation peers who do not.

Survey Instrument and Implementation

questions and should take approximately 10 to 15 minutes to complete. The post-survey

(Appendix H) consists of 46 questions and should take approximately 20 to 25 minutes to

answers based on what they think they should self-report.

14% of students who identify as low-income and/or first-generation to 20% by 2020, so it is

beneficial because it would not provide us with complete information.

20% by 2020 increase.

With assistance from NUs Student Affairs Information Technologys (SAIT)

its pilot stage.

participants believe they feel an increased sense of belonging on campus.

this population as it continues to increase.

To supplement our quantitative approach, we will be conducting a qualitative analysis of

positively influences first-year low-income and/or first-generation students experiences. In this