How We Could Measure Community Impact of Nonprofit Graduate Students' Service-Learning Projects

How We Could Measure Community Impact of Nonprofit Graduate Students' Service-Learning
Projects: Lessons from the Literature

Author(s): Heather Carpenter
Source: Journal of Public Affairs Education, Vol. 17, No. 1 (Winter 2011), pp. 115-131
Published by: National Association of Schools of Public Affairs and Administration (NASPAA)
Stable URL: http://www.jstor.org/stable/27919550
Accessed: 28-08-2014 02:02 UTC
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact support@jstor.org.
National Association of Schools of Public Affairs and Administration (NASPAA) is collaborating with JSTOR to digitize,
preserve and extend access to Journal of Public Affairs Education.
http://www.jstor.org
This content downloaded from 205.221.28.150 on Thu, 28 Aug 2014 02:02:20 UTC
All use subject to JSTOR Terms and Conditions
How We Could Measure Community

Impact ofNonprofit Graduate Students'
Service-Learning Projects:
Lessons from the Literature
Heather
Carpenter
UniversityofSan Diego
Abstract
education scholars are asking the question, "How do
Nonprofit management
we measure
community impact of nonprofit programs?" One way to study the
education is by studying the
community impact of nonprofit management
on the
are
impact that service-learning projects have
nonprofits forwhich they
the
This
literature
that
studies
paper synthesizes
community impact
developed.
of nonprofit graduate students' service-learning projects. This paper also describes
can enhance future
capacity building and evaluation tools and theories that
studies of community impact. Future studies of community impact should
include analysis components from the fields of nonprofit management
education,
service learning, capacity building, and nonprofit evaluation,
consideration specific factors thatmay affect study outcomes.
and take into
A promising new area of practice in nonprofit management

education has
as
of
service
Service
the
introduction
learning.
begun with
learning is defined
"an educational methodology
community service with explicit

for community work, and deliberate
that combines
academic
learning objectives, preparation

reflection" (Gelmon, Holland, Driscoll, Spring, & Kerrigan, 2001, p. v). In
education, students engage in a variety of experiential
nonprofit management
as service
learning projects within local nonprofit organizations, also labeled
learning projects.
has lookedat theeffectsof servicelearning

Although a greatdeal of literature
on the students themselves, fewer studies have measured

the impact of service
or on the communities where the service
learning within nonprofit organizations
was conducted
&
(Perry
Imperial, 2001). This paper reviews literature that studied
the impact of nonprofit graduate
JPAE, 17(1), 115-131
students' service-learning projects on the
Education
JournalofPublicAffairs
115
Impact ofNonprofit Graduate
Community
This paper also reviews capacity

organizations.
evaluation
tools and methods
that can be used
nonprofit
nonprofit
community
Nonprofit
Students' Service-Learning
Projects
building and
in future studies of
impact.
Management
Education
Literature
of the literature on graduate nonprofit management

education focuses
on the
or alumni
development of the field, classroom curriculum, and student
Much
learningoutcomes (Fletcher,2005; Mirabella & Wish, 2000; O'Neill, 2005;

O'Neill & Fletcher,1998;O'Neill & Young, 1988;Wilson & Larson, 2002).
conferences were organized to discuss nonprofit management

education
2005), and scholars have
(Ashcraft, 2007; Burlingame & Hammack,
written articles about the number of nonprofit graduate degree programs (Mirabella,
Several recent academic
2007;Wish & Mirabella, 1998),where theyarehoused, (DobkinHall, O'Neill,

Vinokur-Kaplan,Young,& Lane, 2001; Mirabella & Wish, 2001) and the types
of coursesoffered(Mirabella,2007).
holds that nonprofit management education is important
for educating nonprofit managers. Yet very few studies have evaluated the impact
of nonprofit-focused graduate programs on the nonprofit organizations themselves.
Conventional
wisdom
This
could be done by studying the impact that graduate students' service-learning

on the
are conducted.
In this section,
projects have
nonprofits forwhich they
two studies that
to evaluate the community
of
impact
attempted
nonprofit graduate
students'
service-learning
the overall
community
is also discussed.
are discussed. A
projects
of
several
impact
nonprofit
third study that assessed

graduate
degree programs
first study evaluated the community impact of a "Nonprofit Clinic" at

the University of Pittsburgh's Graduate
School of Public and International Affairs
The
(GSPIA) by studyingthe impactof graduate studentprojectsconducted through

theNonprofitClinic (Bright,Bright,& Haley, 2007). Although theNonprofit
is not connected to a specific nonprofit management

education program,
it engages graduate students in a variety of service-learning projects within the
local nonprofit community. Local nonprofit organizations submit requests for
to receive technical assistance from the clinic. Then clinic staff
proposal (RFPs)
Clinic
to
assigns faculty members and students
semester-long projects.
To assess the impact of these service-learning projects, which they labeled as
technical assistance projects, researchers administered surveys to seven nonprofit
that had received service from the clinic. The survey assessed
organizations
organizational
technical assistance, benefits to receiving the technical assistance,

as a result of the technical assistance, and interest
improvement
improvements
in their organizations
organizations]
focus their attention on the critical issues [that] theywere facing"
satisfaction with
in futureinvolvement
with theclinic.Of therespondents,
86% "[saw] significant
since they received support from the
NonprofitClinic" (Brightet al., 2007, p. 204), and the "serviceshelped [these
116
Journal ofPublicAffairsEducation
Community Impact ofNonprofit Graduate
Students Service-Learning Projects
(p. 204). This studyis importantforunderstandingthecommunityimpactof
student projects because it assessed organizational

the organization as a result of service learning.
to
improvement and benefits
In the second study, researchers at New York University's Robert F.Wagner

School of Public Service assessed the organizational
impact of graduate
students' capstone projects (Schachter & Schwartz, 2009). Forty-two client
were
to determine the impact,
helpfulness, and satisfaction
surveyed
organizations
of student projects. Most of these organizations reported high satisfaction with
Graduate
the studentprojects (4.2 on a scaleof 5) and thehelpfulnessof studentprojects

(3.9 on a scale of 5). The
researchers conducted t-tests to compare satisfaction,

or did not have
and
lasting impact between organizations that did
helpfulness,
as
resource and/or tool
part of the student project. They also found
development
that,of the 13 choices theygave in theirsurvey,thoseprojects that included
an
organizational
assessment
received higher ratings for helpfulness,
lasting
impact,and overall satisfactionthandid thoseprojects thathad includedan
assessment.
organizational
This study is a step forward for the field because it used statistical measures
on
to determine
impact of student projects
nonprofit organizations. The report
of the research also raises some important questions and directions for further
study. For example, what dimensions do satisfaction, helpfulness, and lasting
terms could be broken into more descriptive definitions
impact include? These
to ensure that respondents mean the same

In
thing when giving their ratings.
in
be
of
the
student
could
examined
addition, specific aspects
greater
projects
a more detailed
to determine
of
and
Moreover,
scope
implementation.
degree
depth
a better
understanding of the
presentation of the analyses would have provided
entire sample of respondents
and their perceptions
of student projects.
The laststudy,describednext?although itdid not directlyevaluate the
important because it attempted

impact of student service-learning projects?is
to study the overall community impact of several nonprofit graduate degree
Mirabella andWish (1999) examinednonprofits

programsthroughfocusgroups.
education
involved with nonprofit management
across
at
interviews
the country
focus
group
programs
programs. They
and spoke to faculty, alumni, employers, and funders.They asked these stakeholders
and many
other stakeholders
conducted
to "comment on the impact of the

degree programs" (p. 335). Employers assessed
tools and new collaboration opportunities
students' acquisition of management
with
the university as determinants
of impact.
these
Because relatively
fewfundersparticipated,it isdifficultto interpret
context of impact of nonprofit programs on

nonprofit organizations.
findings in the
The data interpretationwas aimed at faculty and program content. Additionally, the
more on the students'
faculty, alumni, employers, and funder responses focused
skills and abilities as a program outcome and less on specific organizational
andWish described the research as "a first step
impacts. Nevertheless, Mirabella
JournalofPublicAffairsEducation
117
Community
Projects
toward understanding
stakeholders' perspectives" (p. 337). Research linking

and
specific organizational
community outcomes from student service-learning
on what these studies have found.
projects would expand
To review, this section focused on three studies that attempted to study the
community impact of graduate students' service-learning projects. One study
improvements; another study examined
explained satisfaction with organizational
the organizational
satisfaction, helpfulness, and impact of student projects; and
the last study, although itdid not directly evaluate student projects,
explained
that increased collaboration opportunities with the university were
impactful
to
are a step forward for the
nonprofit organizations. Even though these studies
field, more
can be done
ifnonprofit
known
to
study this phenomenon.

education can be considered a
For example,
it is still not
tool for
capacity-building
or if
education
program directors
nonprofit organizations
want their programs to build
within
local
capacity
nonprofit organizations. Some
researchers believe that the nonprofit management
education program's potential
as a
tool for nonprofit organizations

is underappreciated
(Bies,
capacity-building
are
in
Other
researchers
the
field
2008).
community service-learning
starting
to
of
the
the
of
students'
importance
recognize
community impact
evaluating
service-learning
Service-Learning
projects.
Literature
nonprofit graduate students typically engage in service-learning

projects throughout their degree programs, most of the current service-learning
literature focuses on traditional undergraduate
students' attitudes and learning
Although
outcomes
(Killian, 2004;
Imperial, Perry, & Katula,
2007;
Lambright,
2008;
McCarthy & Tucker, 1999;Miller-Millesen& Mould, 2004; Nichols & Monard,
Schumaker, 2005). Little research studies nontraditional

students, or adult
learners engaging in service-learning activities and the impact these activities
have on the nonprofits themselves. The research that focuses on these elements
2001;
asks nonprofits what theywant (Dicke, Dowden, & Torres, 2004).

It does not
on the
or
expand
employer
organizational perspective (Peters, 2009).
The body of literature that has studied the community impact of service
to
learning projects has been slow
develop,
due to several reasons explained
by
Cruz andGiles (2000). First, it isdifficultto define thecommunity.Is it the
of the nonprofit organizations, or the

problems in evaluating the community
are
of
There
impact
service-learning projects.
disagreements about what variables
an
to
or
are the
in
what
study
organization
community,
appropriate methods for
data collection, and what analyses of the data would be useful. Notwithstanding
the constituents
nonprofit organizations,
funders? Second, there are methodological
researchers are beginning to attempt to study the community

a few methods and studies that
service
impact of
learning. This section describes
were
to address the
developed
challenges described earlier.
these concerns,
118
Both qualitative
and quantitative methods
have been used to study the
projects (Jorge,2003; Schmidt& Robby,

communityimpactof service-learning
is
work ofClarke (2000). Clarke created
the
One
mixed
methods
2002).
study
the3-1model as a tool foruse by nonprofitsand evaluators in studyingthe
community impact of service-learning projects. The 3-1 model framed community

as a process and addressed the question, "How should an evaluation
impact
model be designed in order to acknowledge, measure and reflect the dynamic
community impact of service initiatives and specific service-learning programs?"
a focus group, observations,

(p. 162). Clarke created themodel using interviews,
and a survey The 3-1 model evaluated community impact in three parts.
Initiator(ThefirstI}. The storyof planningand goals of the service

learning project from the perspective
of the service-learning
initiator.
Initiative(The secondI): The storyof implementation

andwhat
activities took place during the service-learning project from the

perspectives of the organization and university.
Impact (The thirdI): The storyof impact,includingthe resultsof
service-learning project and its impact in the community from the

perspectives of the organization and university, (p. 163-164)
towork, an organization must also understand its "good
indicators, utilization of the project, method of change and inclusion of
For the 3-1 model

practice
community
a
impact"
(p. 163).
Clarke testedthismodel in thecommunityforvalidityand reliability

using
and quantitative methods. The model met its purpose of
measuring, and reflecting community impact" (p. 165). For
variety of qualitative
"acknowledging,
example, themodel measured
the success of community and campus partnerships

and found universities were vital in helping nonprofits utilize the 3-1 model
(Clarke,2003).
The model was created using utilization-focused evaluation, which is evaluation

an
important
developed from the perspectives of many stakeholders. This study is
step away from foundations' perspectives of nonprofit evaluation and toward

foundations evaluate nonprofits on the
nonprofits' view of evaluation. Whereas
to show their
and impact, nonprofits use a variety of methods
For example, Carman
(2007) stated, "Some community-based
accomplishments.
or
not
understand
organizations do
distinguish between reporting, monitoring,
and management
practices and evaluation" (p. 72).
Other researchers also agree with the nonprofit view of evaluation. In fact,
basis of outcomes
the effectiveness of service-learning projects based on successful

some
university and nonprofit partnerships, and
provide best practices for
some evaluate
& Batholomew,2006; Bringle

and communitypartnerships(Basinger
university
& Hatcher, 2002; D'Agostino, 2008; Ikeda,Cruz, Holland, Rice, & Sandy,
119
Community
Students7 Service-Learning
Projects
2007; Jacoby2003;Worrall, 2007). Dorado andGiles (2004) assessed successful
terms of continued
partnerships in
partnership between nonprofits and universities,
more than two semesters.
specifically partnerships that lasted
They described
the development
process of successful partnerships
from both the nonprofit and
university perspective.
Gelmon (2003) developed a frameworkforassessingthecommunityimpact
of service learning from the organizational perspective. This framework included

involvement in the service-learning partnership
assessing how the organizations
influencedits"(a) capacityto fulfill

mission; (b) economicbenefits;
organizational
(c) social benefits; (d) nature of community-university partnership;
(e) satisfaction
with partnership;
of partnership"(pp. 52-53). This framework
and (f) sustainability
is part of a larger assessment model
in which evaluators could assess the
and university impact. Gelmon
cautioned evaluators using the
organizational
entire model, saying, "Individuals designing and collecting the assessment must
be cautious to avoid criteria thatmay be interpreted as a performance review of
a
the community organization"
(p. 56). This shows there is fine line between
assessment
and capacity-building
evaluation
efforts.
Bacon (2002) assessednonprofitand universitypartnershipsby analyzing
how each group viewed learning and knowledge. Through focus group interviews
of four service-learning professors and four nonprofit employers, Bacon found
that these groups viewed the impact of service learning differently. The study
reported that nonprofits and universities had disparate views of knowledge
and learning. For example, they did not agree on the project's needs
assessment. Universities assumed the
nonprofits would tell the students what
to
the
needed
they
service-learning project
accomplish. Nonprofits wanted guidance
from students to help them determine the needs the projects would address.
creation
focus group research contributed to helping community organizations

and universities understand the benefits of collective learning during the service
Bacon's
learning process.
In anotherpartnershipstudy,
almost 100
SandyandHolland (2006) interviewed
community partners about their experiences in partnering with eight California

universities, using focus groups. The community partners assessed impact in
terms of "client outcomes and sustaining and
capacity"
enhancing organizational
in
undertook
students
that
would
35).
projects
(p.
Additionally,
organizations
not otherwise
have been completed by the organizations. The authors did not

some
information that would have informed this review, including the
report
were traditional or adult learners.
service
of
types
projects and if the students
Slightlybeyond the scope of partnerships,one studyevaluatedadult learners
a
course
engaging in service-learning projects. Faculty in Public Administration
at
for
identified local nonprofit and government organizations
Troy University
studentsto conduct projects in (Walder& Hunter, 2008). At the end of the

semester, representatives
120
from these organizations
attended
the student
presentations. The client organizations provided feedback to the students

about their presentations. Researchers noted there was an unexpected benefit
to the
in this process. During
that participated
the student
organizations
presentations,
organizations
met
representatives.
organizational
In research that contributed
and made
lasting connections
with
other
to an
one
understanding of community impact,
researcher evaluated the efforts of adult learners in a nonprofit-focused graduate
degree program that engaged in service-learning projects (Bushouse, 2005).
of 11 organizations were interviewed using questions focused on three

areas: the
service-learning relationship between the student and organization,
the organizations attitude toward continuing in the service-learning relationship
Members
main
with
the university, and the utilization
of the project within
the organization.
Overall, thenonprofitsinBushouse s studyreportedgreatutilityof the
service-learning projects. Most of the organizations

implemented
in some manner. The organizations
recommendations
conveyed
the students'
that itwas
for them to engage in the RFP process from the university. This
process allowed them to gauge the opportunity cost of the project ahead of time.
Bushouse designed her study based on Enos and Morton's
(2003) framework for
beneficial
on relations between
partnerships, which focuses
developing campus-community
the university and the community. As a result, the study did not discuss service
as a
tool, and Bushouse stipulated that the research
capacity-building
learning
therewas difficulty defining the community
could not be generalized. Moreover,
because nonprofits did not equal the community but were an important part of
the community.
Even though the service-learning literature is beginning to focus on the
community impact of nonprofit graduate students' service-learning projects,
see a need for further

development of thiswork. Ward and Wolf-Wendell
(2000) investigated thiswork by looking forways community organizations and
universities can partner more effectively in service-learning projects. They asked,
many
"How do we get campuses to do with their communities rather than do for

them?" (p. 774). They proposed the following recommendations:
Connect
through commonalities.
Blur boundaries between campus and communities.
Consider
the position, history, and the power or powerlessness
involved in the service relationship.
assessment.
Encourage
reciprocal
Rethink service missions to include and reward public
of all
service and
genuine communitypartnerships,(pp. 774-776)

have face validity, they are not backed
these recommendations
Although
to support service
is
evidence.
There
also
little
by
empirical evidence
learning
121
Community
as a
tool for nonprofits, as evidenced

potential capacity-building
articles asserting service learning as a capacity-building
tool.
Projects
by the lack of
section reviewed several studies within
the field of service learning

that attempted to measure community impact of nonprofit graduate students'
projects. As noted earlier, although scholars in the field of service learning have
are
to include
impact, studies
community
beginning
difficulty measuring
In one article, the process of community impact
community impact components.
was studied; and in several other studies, community impact was measured
in
terms of successful
In
another
benefited
yet
partnerships.
study, organizations
This
opportunities with other organizations during the student

In the last study described earlier, organizations expressed
project presentations.
utility from the student projects and implemented project recommendations.
measure the
Many of these studies did not directly
community impact of nonprofit
from the networking
students' service-learning projects, but they represent a step forward in

understanding how universities and nonprofit organizations partner and view
more work needs to be done
community impact. Researchers recognize that
graduate
to evaluate
the community and organizational

impact of these programs. One
researcher said, "In hindsight we should have employed a more formal mechanism
for securing feedback from our community partners" (Lowery, 2007). This is
where researchers in nonprofit management
education and service learning can
in
utilize best practices tools and methods

evaluation to study community impact.
Capacity-Building
Capacity
Tools
building,
and nonprofit
capacity-building
and Methods
a well-known
term in the
nonprofit
sector, is linked to
improvingorganizationalperformance.
Wing (2004) defined capacitybuilding
as
the ability of an organization to fulfill itsmission"

(p. 155). Since
can be
to define
organizational missions differ, it
challenging
specific types of
tomission fulfillment. Backer, Miller, and
activities
that
lead
capacity-building
or "interventions,"
Bleeg (2004) described three types of capacity-building activities,
interventions are (a)
that would
lead to overall nonprofit effectiveness. These
to
undertake the kinds of internal changes capacity
assessment, or, "readiness
"increasing
buildingwill require"(p. 3); (b) technicalassistanceand organizational

development
consultation
from the outside;
(c) direct financial support from funders or

efforts.These three interventions need to occur
and
for capacity-building
in order for nonprofits
simultaneously
donors
to
perform effectively.
The challengewith theproposed interventionsis thattheydo not includea
strategy formeasuring
nonprofit effectiveness, which
is difficult to do. Herman
withmeasuringnonprofiteffectiveness;
andRenz (2008) described
many difficulties
one
is that nonprofit effectiveness ismultidimensional.

They argued that there is
measure or group of indicators for
effectiveness.
This same
single
nonprofit
argument can be used with nonprofit performance; as with nonprofit effectiveness,
no
122
there isno consensus on standard performance measures for nonprofit organizations

are related as well
(Wing, 2004). Since nonprofit effectiveness and performance
can
to
as linked to
it
be
evaluate
efforts,
capacity
capacity-building
challenging
as well. However, researchers are
towork around the
efforts
starting
building
complexities of nonprofit effectiveness and nonprofit performance by creating
measures
to evaluate
efforts.This section provides examples
capacity-building
of three proposed methods of such efforts that can be used to evaluate efforts
within nonprofit graduate students' service-learning projects.
Wing
challenges
(2004) addressed capacity building and nonprofiteffectiveness
through
How
seven
questions:
[can we measure
capacitybuilding] ?
How
can we measure
such] an abstract concept

performance
measure
[as nonprofit
we cannot
improvement when
performance?
we measure
whose
improvement?
Against
goals should
can be done about unrealistic timetables for both
What
capacity
building and itsevaluation?
How
can we document
Should we measure
how soft people relate to hard systems?

behavioral change or clients'
participants'
internal
How
learning?
a
can researchers
study when
design
what theyareworking on? (pp. 154-159)
consultants
keep changing
Wing also argued that itcould be challengingto actually evaluate the
if stakeholders have disparate goals

efforts of an organization
capacity-building
non
For
for measuring
improvement.
example, if the funder, consultant, and
were not to reach
efforts
goal consensus, capacity-building
profit organization
would not be aligned.AlthoughWing (2004) expressedthebelief thatnonprofit

a systems
to
tomeasure, he nonetheless
proposed
approach
performance is difiScult
efforts. In this approach, evaluators must understand
evaluate capacity-building
the relationship and balance of the system of the organization and the people in
it. By evaluating people's ability to engage in the evaluation process and by
an
to handle
efforts,
capacity
evaluating
capacity-building
organization's
researchers and practitioners can create more effective capacity-building
tools.
This is something that researchers could attempt to do in assessing service-learning
projects.AlthoughWing's (2004) assertionsareplausible, theyare not backed by

empirical
through
evidence. Another method
a model
for evaluating capacity-building
efforts is
approach.
Connolly andYork (2002) proposed a logicmodel forevaluatingcapacity

ofwhy
They definea logicmodel as "a pictorialrepresentation
building efforts.
a
effort
will happen" (p. 37). This logicmodel consists
and how capacity-building
Education
123
Community
Projects
of inputs, also known as resources employed; activities thatwill occur during

the evaluation; outputs, also known as results; and outcomes or changes in the
program.Connolly andYork addressedall the issuesthat

Wing proposed.Wings
and Yorks assertions would be bolstered by empirical evidence.

and Connolly
Scale (Poole, Nelson, Carnahan,
The Program Accountability Quality
a
was
as
measurement
&
created
scale for
Tubiak,
2000)
performance
Chepenik,
assertion
that performance
efforts.The scale challenges Wings
capacity-building
cannot be measured
logic model
and consists of seven categories, many similar to those in the

and York (2002). These are resources, activities, outputs,
of Connolly
goals, indicators, and evaluation plan. In developing the scale, the

authors attempted to find similarities among nonprofit organizations and how
they defined capacity-building activities. The authors addressed interrater reliability
outcomes,
cases to 3 different evaluators for

rating, and they
by randomly assigning 20
further testing of
achieved an interrater reliability coefficient of .84. Through
interrater reliability, the researchers eliminated some items in the tool that were
set of items. The

was intended
strongly associated with the total
resulting scale
to
identify their technical assistance needs and may provide
help organizations
the impact of nonprofit management
education and
insight into measuring
not
to be
service-learning projects. The authors stipulate that further research needs
test
to
how the scale might be generalized and used sector-wide. A
conducted
variety of other factors may affect the validity of this performance management
scale, such as organizational
size and culture.
efforts
This sectionhighlightedthechallengesof evaluationcapacity-building
and nonprofit effectiveness. The studies reviewed earlier provide alternative

a
to
a systems
approaches
approach,
evaluating capacity-building efforts through
scale. These studies suggest how
model approach, and a performance management
one
might go about measuring community impact of nonprofit graduate students'
to take
to
study
place
service-learning projects. More empirical research needs
the commonalities among nonprofit organizations and how they define capacity
building methods and organizational effectiveness. Some researchers stipulate that
to become better consumers of

nonprofits need
capacity-building efforts (Millesen
& Bies, 2007). Researchers within the field of nonprofit evaluation are starting to
understand how nonprofits relate to and respond to community impact studies.
Nonprofit
Evaluation
Tools
and Methods
to
Nonprofit organizations
struggle with conducting evaluation activities
effectiveness and community impact. The emerging
show their organizational
field of nonprofit evaluation has expanded in recent years due to a push from
funders. In fact, a special issue of the journal New Directions for Evaluation was
to this
researchers attempt
topic (Carman & Fredericks, 2008a). When
must
of
students'
study community impact
projects, they
nonprofit graduate
understand the history of evaluation effortswithin specific nonprofit organizations,
devoted
to
124
Students Service-Learning
Projects
because a community impact study is a form of an evaluation. Three specific studies

are described to show how
to evaluation efforts.
nonprofits understand and relate
In one study, Carman
how nonprofits
and Fredericks
assessed
(2008b)
engaged in evaluation activities. The nonprofits included in their study conducted
written observations, face-to-face interviews, surveys, and focus groups; they col
lected evaluation data on program expenses, people served, client demographics,
satisfaction levels, and/or program outcomes and results. Logic models, a popular
tool for professional evaluators, were not used by these nonprofits. Many
as "a resource drain and
nonprofits included in this study viewed evaluation
an external,
tool"
tool, and as a strategic management
promotional
is
how
This
for
important
position
nonprofits may
(pp. 58-59).
understanding
to community impact studies.
respond
distraction,
In a second study, Carman

reported that funding agencies affected the scope
funders requested
and capacity of nonprofit evaluation activities. Although
to conduct evaluations, very few

nonprofits
provided funding and support for
not
utilize the evaluation data in their
evaluations.
funders
such
did
Moreover,
fundingdecisions (Carman, 2009). Funders at thefederal levelare pushing
to do more evaluation, but those at the state and local level are not.
nonprofits
This stance affects nonprofits because those that receive federal funding are engaging
in evaluation activities as required by their funders.
Since nonprofits may not understand the type of evaluation work that the
community impact study is trying to achieve, researchers must take this situation
into consideration when attempting to study the community impact of nonprofit
graduate students' service-learning projects. For example, when researchers go
into organizations using evaluation mechanisms

such as surveys or interviews,
not have
engaged in this type of evaluation work before.
nonprofits may
The final study that focused on the evaluation capacity of nonprofit
organizations
is relevant because
nonprofits regularly struggle with evaluation
fundingand capacity issues.Alaimo (2008) usedVolkov and King s (2007)
as well as action steps of executive directors

capacity-building checklist
to assess evaluation capacity within nonprofit organizations. He reported that
executive directors who could balance internal organizational contexts as well as
evaluation
external pressures, like pressures from funders, had longer-term evaluation capacity.
were more
to
prioritize evaluation efforts and to embed
organizations
likely
evaluation in the organizational culture.
These
The
evaluation
capacity
issue is important because nonprofits may not have

in studies of community impact. Additionally, most of
the capacity to participate

the community impact studies described
in this article focused on the partnership

aspect, and very few assessed the impact of student work on nonprofit operations.
It isunknown ifthe lackof studiesthatassessed the impactof studentwork
relates to the capacity of nonprofits to participate in such assessments or ifother

factors are at play. Despite
these considerations, researchers must understand
125
Community
issues within
Projects
when
conducting community
impact studies.
The preceding section described nonprofit evaluation activities, funding
support for nonprofit evaluation, and evaluation capacity issues. These studies
capacity
indicate the difficulties researchers may encounter when

impact of graduate students service-learning projects.
for Future
Recommendations
studying the community
Research
section integrates the lessons learned from the nonprofit management

for
and service-learning literature and provides recommendations
This
education
future studies of community impact of nonprofit graduate students service

com
are divided into two
categories,
learning projects. These recommendations
munity impact analysis components and factors to take into consideration.
Analysis
of community
include these components:
impact should
The ProgramAccountabilityQuality Scale developed by Poole and

his colleagues
dent projects
tomeasure
(2000)
impact of nonprofit graduate
stu
mechanisms
that nonprofit organizations may under
stand (e.g., written observations, face-to-face interviews, surveys, and
focus groups)
Evaluation
The ways
inwhich
building
tool within
Service-learning
service learning might be used as a capacity
models and frameworks, such as the 3-1 model
(Clarke,2000, 2003) and Gelmons framework(2003)
Factors
to consider
Recognize
in future studies of community
that capacity may

studies.
impact
Understand
dimensions
learners are nontraditional
be a major
impact:
factor in community
learners in studies of impact. Adult

students, and most nonprofit management
of adult
cater to these adult

working professionals.
Create partnerships with nonprofit evaluation scholars to complete
joint studies of community impact.
education
programs
Work with capacity-building

scholarstodevelop empirically
built logic
models
in
studying the community
impact of service-learning projects.
Conclusion
This
article reviewed
students
literature that studied the impact of nonprofit graduate

on the
projects
nonprofits themselves. It is clear from
service-learning
the studies described earlier that researchers have not yet bridged the relationships
126
across the
most scholars focus on just one area
(nonprofit
disciplines, because
management education, service learning, capacity building, or nonprofit evaluation).
The studies described earlier have helped define and increase our understanding
of service learnings contributions to nonprofit organizations, and more research
isneeded topinpoint specificcontributions.In addition to thecapacitybuilding
and nonprofit evaluation tools and methods described earlier, further research
should also inventory the types of service-learning activities that nonprofit
students conduct within
graduate
the nonprofit community. Mirabella
and Renz
(2001) began thiswork with a studyof theoutreach activitiesof nonprofit
education programs. Future research on service learning in nonprofit

education programs can expand on thiswork by inventorying and
at
the service-learning components
graduate-level
nonprofit
across
the
education programs
United States and then comparing
management
management
categorizing
management
these components
across programs.
References
Alaimo,
S. P. (2008).
Nonprofits
Ashcraft, R.
and evaluation: Managing

expectations
Nonprofits
and evaluation: New directions for evaluation, 119, 73-92.
(Ed.)
Benchmark
(2007).
ic studies.
Nonprofit
from Human
Retrieved
third decennial
conference
on
nonprofit
and philanthrop
and Voluntary Sector Quarterly, 36.
A. N., &
Backer, T. E., Miller,
3: The
from the leaders perspective.
on
perspectives
nonprofit capacity-building.
Institute http.7/v/ww.
wvdo-or.org/uploads_wvdo/
E. (2004).
Bleeg, J.
Interaction Research
Donor
resources/Non-Profit_Capacity_Building.pdf
Bacon, N.
(2002).
Basinger,
.,&
Bartholomew,
expectations,
Bies, A.
in faculty and community

Service Learning, 9, 34-44.
Differences
nal of Community
. (2006).
and outcomes. Michigan
partners'
Jour
in nonprofit organizations: Motivations,

Service-learning
Journal
theories of learning. Michigan
of Community
context and
education: Considerations
(2008, August). Nonprofit management
capacity. Paper
of
at the annual
ofManagement,
CA.
Anaheim,
meeting of the Academy
presented
& Haley, L. L. (2007). Nonprofit

outreach services: Using outreach to increase
a
to
and
educational
capacity
provide
experience for students. Journal ofPublic
quality
Bright, L., Bright, C,

nonprofits'
Education, 13, 335-343.

Affairs
& Hatcher,
J.A. (2002). Campus
Bringle, R. G.,
ment. Journal
Social
Issues, 58, 503-516.
of
&
community
partnerships:
The
terms of engage
A summary
Burlingame,D., & Hammack, D. (Eds.). (2005). Educationfor a civilsociety:
ofthe2004
conference. Indianapolis:
Bushouse,
.K.
(2005).
Indiana University, Center
Community
on
Philanthropy.
and service learning: Resource
constraints
to
with universities.
ServiceLearning,12, 32-40.
MichiganJournalofCommunity
buildingpartnerships
127
Community
J. G.
Carman,
Evaluation
(2007).
reality. American
-.
Journal
among
practice
community-based
28, 60-75.
ofEvaluation,
hinders, and evaluation:

(2009). Nonprofits,
Public Administration,
39, 374-390.
J.G., &
Carman,
Fredericks, K. A.
gram
(2008a).
(Eds.).
119. San Francisco:
evaluation,
Research
organizations:
in action. American
Accountability
and evaluation: New
Nonprofits
Projects
into the
Review
of
directions for pro
Jossey-Bass.
and evaluation: Empirical

evidence from the
J.G., & Fredericks, K. A. (2008b). Nonprofits
field. In J. G. Carman & K. A. Fredericks (Eds.),
Nonprofits and evaluation: New directions in
119. San Francisco: Jossey-Bass, 51-71.
evaluation,
Carman,
Clarke, M. M.
(2000).
the community
Evaluating
impact of service initiatives: The 3-1model
(Doctoral
dissertation,PeabodyCollege forTeachers ofVanderbiltUniversity,2000). ProQuestDigital Dis

sertations. AA
-.
(2003).
9991711.
in
the community
Finding
(Eds.), Deconstructing
J. Eyler
3-21).
Nashville:
P., & York. P. (2002). Evaluating

Connolly,
capacity-building
nizational Devebpment
Practitioner, 34, 33-39.
. I., &
Cruz,
Journal
E.
(2000). Where
Service Learning,
of Community
M.
D'Agostino,
D.
Giles,
J. (2008).
Fostering
s the
community
28-34.
SI2000,
civically
Dicke,
S., & Torres,
L., Dowden,
J. (2004).
10, 199-208.
Public Affairs Education,

Dobkin
Hall,
depends
grams
service
research? Michigan
service-learning
university
learning:
Orga
and service learning.
A matter
of ideology. Journal
of
M., Vinokur-Kaplan,
D., Young, D., & Lane, F. S. (2001). Where
you stand
sit: The
location for university-based
pro
implications of organizational
in nonprofit management.
Public Performance & Management
Review, 25, 74-87.
S., & Morton,
you
D.
nal ofCommunity
Enos,
in
organizations.
P., O'Neill,
on where
S., & Giles,
Dorado,
Successful
efforts for nonprofit
society: The
engaged
Education, 14, 191-204.

In S. H.
&
3-'T model.
Billing
Information Age Publishing.
research: The
service-learning
service-learning(pp.
E. (2004). Service-learning
Service Learning, 77,25-37.
K.
(2003).
Developing
partnerships:
Paths of engagement. Michigan
theory and practice of campus-community
Jour
partnerships.
In
B. Jacobyet al. (Eds.),Buildingpartnerships

for servicelearning(pp.20-41). San Francisco:
Wiley.
Fletcher, K.
(2005).
professional
Gelmon,
S. B.
a master's
in
The
impact of receiving
degree
lives. Nonprofit and Voluntary Sector Quarterly, 34, 433-447.
(2003).
as a means
Assessment
of building
service-learning
partnerships.
on
graduates'
In B. Jacoby et
al. (Eds.). Buildingpartnerships

(pp. 42-64). San Francisco:
Wiley.
for service-learning
Gelmon,
S. B., Holland,
.A., Driscoll,
and civic engagement: Principles

Herman,
R. D., &
theory: Nine
128
Renz,
D. O.
and
A.,
Spring, A., &
effectiveness
Advancing
nonprofit organizational
and Voluntary Sector Quarterly, 18, 399-415.
(2008).
theses. Nonprofit
Kerrigan, S. (2001). Assessing

RI: Campus
Compact.
service-learning
techniques. Providence,
research and
Projects
CommunityImpactof
NonprofitGraduateStudentsService-Learning
.,& Sandy, M.
(2007). Community voices: A California
San Francisco: California Campus
Compact.
campus
service learning into public

T., Perry, J. L., & Katula, M. C. (2007).
Incorporating
Lessons from the literature. Journal ofPublic Affairs Education,
13, 243-264.
affairs
.,Holland,
Ikeda, E., Cruz,
.,Rice,
compact study on partnerships.

Imperial, M.
programs:
Jacoby, B.
Jorge, E.
(Ed.).
(2003).
Building
Outcomes
(2003).
partnerships for service-learning.
for community
San Francisco:
in an unmediated
partners
ServiceLearning,10,28-38.
Michigan JournalofCommunity
Killian,
J. (2004).
Education, 10, 209-224.

techniques.
Lowery, D. (2007). Community-based
quality of life indicators: A service-learning
ate statistics class. Journal
Public
13, 425-438.
of
Affairs Education,
Lambright,
K.
Lessons
(2008).
outside
of the classroom:
program.
service-learning
traditional, distance,
Combined,
experimentation:
Pedagogical
Jossey-Bass.
and service-learning
in a
exercise
gradu
the effectiveness of service-learning
Examining
Education, 14, 205-217.

projectsat achievinglearningoutcomes.JournalofPublicAffairs
McCarthy,
A. M.,
&: Tucker, M.
implementation.
(1999).
Management
of
Student
attitudes
Education,
23,
toward service-learning:
for
Implications
554-573.
in
D. H.
(2004). Project-based
J. L., & Mould,
learning
from an educational
partnership between the United States and Kyrgyzstan.
Miller-Millesen,
education:
Journal
L.
Results
Education, 20,247-258.
Who isdoingwhat forwho and to

Millesen, J.L., &: Bies,A. L. (2007).Nonprofitcapacitybuilding:
what end?Journal
Management, 11, 18-27.
forNonprofit
Mirabella,
R. M.
Mirabella,
R. M.,
educational
(2007). University-based
A
review
and
studies:
10-year
projections
thropic
Quarterly, 364, 11S-27S.
&
Renz, D. O.
of institutional mission
Mirabella,
R. M.,
programs:
and philan
programs in nonprofit management
of future trends. Nonprofit and Voluntary Sector
& Wish,
Perspectives
outreach
(2001). Nonprofit management
and setting. Public Performance & Management
N.
B.
(1999).
of multiple
Educational
stakeholders.
programs: An
Review, 25,
impact of graduate
Nonprofit Management
examination
14-29.
nonprofit degree
and Leadership,
9,
329-340.
-.
(2000).
managers.
-.
(2001).
An
Nichols,
updated
of graduate
The "best place" debate: A comparison
Public Administration Review, 60, 219-229.
programs
for nonprofit
of nonprofit organizations:
educational programs in the management
University-based
census of U.S. programs, Public
Review, 25, 30-41.
Performance and Management
A. H., & Monard,
student characteristics.
K.
(2001).
Educational
M.
(2005). Developmental
ment and
16, 5-17.
Leadership,
O'Neill,
education
Designing
intergenerational
Gerontology, 27, 37-48.
service-learning
contexts of
education.
courses based
on
Nonprofit Manage
129
&
M.,
O'Neill,
U.S.
education:
management
Nonprofit
and world
perspectives.
Praeger.
D.
&c Young,
M.,
. (1998).
Fletcher,
CT:
Westport,
O'Neill,
Students' Service-Learning Projects
Community
R.
(1988).
managers
Educating
of nonprofit organizations. New
York:
Praeger.
Perry, J. L., &

Imperial, M. T. (2001). A decade of service-related research: A map of the field. Non
R. A. (2009). Using focus groups and
and
Voluntary Sector Quarterly, 30, 462-479.Peters,
profit
stakeholder surveys to revise theMPA
curriculum. Journal ofPublic Affairs Education,
15, 1-16.
S., Chepenik, N., &Tubiak,
C,
(2000). Evaluating
J., Carnahan,
performance
in
systems
quality scale. American
nonprofit agencies: The program accountability
Poole, D., Nelson,

measurement
Journal
& Holland,
Sandy, M.,
spectives
13, 30-43.
D.
Schachter,
on
15-26.
21,
ojEvaluation,
B. A.
campus-community
R., &
D.
Schwartz,
and common
Different worlds
(2006).
partnerships. Michigan
(2009).
The
value
Journal
of capstone
Journal
Schumaker,
A.
Robby, M.
of Community
A. M.
(2002). What's
Service Learning,
the value
of service-learning
participating
client agen
to the community?
Michigan
9, 27-33.
In search of a model
(2005).
on
projects
cies.JournalofPublicAffairs
Schmidt, A., &
partner per
ground: Community
of Community Service Learning
for effective group projects:
From
theMPA
student
perspective.
.B., &
A checklist for
King, J.A. (2007).
building organization
from http://www.wmich.edu/evalctr/archive_checklists/ecb.pdf
Volkov,
L. S., & Hunter,
Walder,
D.
(2008).
Public Affairs Education,
to
I., &
.T.
in service
learning. Journal
(2000).
Behavioral
service learning: Moving

Community-centered
Scientist, 43, 767-780.
R. S. (2002). Nonprofit management

students: Who
and Voluntary Sector Quarterly, 31, 259-270.
Larson,
enroll. Nonprofit
Wing,
L.
doing with. American

M.
Wilson,
courses: Variations
(2004).
capacity. Retrieved
of
14, 219-239.
K., & Wolf-Wendell,
Ward,
Client-based
evaluation
Assessing
the effectiveness of capacity-building
they
initiatives: Seven
from doing
are and
why
for
they
issues for the field.
SectorQuarterly,33, 153-160.
Nonprofitand Voluntary
. B., & Mirabella,
Wish,
R. M.
(1998).
programs. Nonprofit Management

Worrall,
L.
Journal
130
(2007).
Asking
of Community
Curricular
variations
and Leadership,
in nonprofit management
graduate
9, 99-109.
the community: A case study of community

partner perspectives. Michigan
Students' Service-Learning Projects
is a PhD candidate at University of San Diego, where she

Carpenter
serves as a Research Associate
in the Caster Family Center forNonprofit and
Heather
Philanthropic
Research. Heathers
experiential education,
nonprofit
research interests are nonprofit education,

leadership, nonprofit workforce issues, and
She authorsthenationallyknownblog,NonprofitLeadership
nonprofittechnology.
601 (http://www.nonprofitalternatives.org/page/blog).
131

How We Could Measure Community Impact of Nonprofit Graduate Students&#39; Service-Learning Projects

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How We Could Measure Community Impact of Nonprofit Graduate Students&#39; Service-Learning Projects

Uploaded by

Copyright:

Available Formats

How We Could Measure Community Impact of Nonprofit Graduate Students' Service-Learning

Projects: Lessons from the Literature

How We Could Measure Community

and take into

A promising new area of practice in nonprofit management

community service with explicit

learning objectives, preparation

has lookedat theeffectsof servicelearning

on the students themselves, fewer studies have measured

JPAE, 17(1), 115-131

students' service-learning projects on the

Impact ofNonprofit Graduate

This paper also reviews capacity

of the literature on graduate nonprofit management

learningoutcomes (Fletcher,2005; Mirabella & Wish, 2000; O'Neill, 2005;

conferences were organized to discuss nonprofit management

2007;Wish & Mirabella, 1998),where theyarehoused, (DobkinHall, O'Neill,

could be done by studying the impact that graduate students' service-learning

third study that assessed

first study evaluated the community impact of a "Nonprofit Clinic" at

(GSPIA) by studyingthe impactof graduate studentprojectsconducted through

is not connected to a specific nonprofit management

technical assistance, benefits to receiving the technical assistance,

focus their attention on the critical issues [that] theywere facing"

NonprofitClinic" (Brightet al., 2007, p. 204), and the "serviceshelped [these

Community Impact ofNonprofit Graduate

Students Service-Learning Projects

(p. 204). This studyis importantforunderstandingthecommunityimpactof

student projects because it assessed organizational

In the second study, researchers at New York University's Robert F.Wagner

the studentprojects (4.2 on a scaleof 5) and thehelpfulnessof studentprojects

researchers conducted t-tests to compare satisfaction,

that,of the 13 choices theygave in theirsurvey,thoseprojects that included

received higher ratings for helpfulness,

impact,and overall satisfactionthandid thoseprojects thathad includedan

to ensure that respondents mean the same

and their perceptions

The laststudy,describednext?although itdid not directlyevaluate the

important because it attempted

Mirabella andWish (1999) examinednonprofits

to "comment on the impact of the

the university as determinants

context of impact of nonprofit programs on

Impact ofNonprofit Graduate

stakeholders' perspectives" (p. 337). Research linking

study this phenomenon.

tool for nonprofit organizations

nonprofit graduate students typically engage in service-learning

Imperial, Perry, & Katula,

McCarthy & Tucker, 1999;Miller-Millesen& Mould, 2004; Nichols & Monard,

Schumaker, 2005). Little research studies nontraditional

asks nonprofits what theywant (Dicke, Dowden, & Torres, 2004).

due to several reasons explained

Cruz andGiles (2000). First, it isdifficultto define thecommunity.Is it the

of the nonprofit organizations, or the

researchers are beginning to attempt to study the community

Community Impact ofNonprofit Graduate

and quantitative methods

Students Service-Learning Projects

have been used to study the

projects (Jorge,2003; Schmidt& Robby,

community impact of service-learning projects. The 3-1 model framed community

a focus group, observations,

Initiator(ThefirstI}. The storyof planningand goals of the service

How We Could Measure Community Impact of Nonprofit Graduate Students' Service-Learning Projects

How We Could Measure Community Impact of Nonprofit Graduate Students' Service-Learning Projects