New York City Center For Economic Opportunity: An Evidence-Based Approach To Alleviate Poverty

HKS724
Case Number 1971.0
New York City Center for Economic Opportunity: An Evidence-Based

Approach to Alleviate Poverty
Certitude is not the test of certainty.

We have been cocksure of many things that were not so.
Oliver Wendell Holmes, Jr., Natural Law, 1918
Introduction
“Today, I am committing to a major reduction in the number of children, women, and men who live in
poverty in this city,” declared New York City’s Mayor, Michael Bloomberg in February 2006 as he announced the
1
creation of the Commission for Economic Opportunity. The 33 member, public-private Commission had been
charged by the mayor to identify innovative approaches that would reduce poverty and increase access to oppor-
2
tunity in New York. After six months of deliberations, the Commission presented a list of actionable recommenda-
tions. The Commission’s report to Bloomberg included specific initiatives to tackle the stubbornly high rates of
poverty among working poor adults, young adults between the ages of 16 and 24, and families with children aged
3
five and below—groups they believed were representative of “the many faces of New Yorkers living in poverty.”
Perhaps the most novel aspect of the report was the section on implementation. It emphasized “account-
ability and measurement” as critical factors in the success of initiatives. Specifically, the Commission urged the
Bloomberg administration to devise a new measure of poverty in New York and to conduct “rigorous evaluations
4
of initiatives.”
Veronica White was closely following the work of the Commission in 2006. With a Juris Doctorate from
Harvard Law School and a Master of Arts in Law and Diplomacy from Tufts University, White had first worked in
New York City government under Mayor Koch’s administration in the 1980s, and had gone on to hold executive
1
De Mause, Neil “Target of Opportunities,” City Limits Investigates, Spring 2008, Vol. 32, No. 01.
2
The Commission comprised representatives from government, business, nonprofit and academic spheres. The chairpersons
were Geoffrey Canada, founder of the acclaimed Harlem Children’s Zone and Richard Parsons, chairman of Time Warner Inc.
3
“Increasing Opportunity and Reducing Poverty in New York City,” Commission for Economic Opportunity Report to Mayor
Bloomberg, September 2006, p. 12.
4
The U.S. Federal Poverty measure used since 1969 was calculated as three times the cost of a basic food plan, adjusted annu-
ally for inflation.
This case was written by Anjani Datla, Case Writer, for Julie Boatright Wilson, Harry Kahn Senior Lecturer in Social Policy, and
Dan Levy, Senior Lecturer in Public Policy at the John F. Kennedy School of Government (HKS), Harvard University. HKS cases
are developed solely as the basis for class discussion. Cases are not intended to serve as endorsements, sources of primary
data, or illustrations of effective or ineffective management. (January 2012)
Copyright © 2012 President and Fellows of Harvard College. No part of this publication may be reproduced, revised, translated,
stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means without the express written
consent of the Case Program. For orders and copyright permission information, please visit our website at
http://www.ksgcase.harvard.edu/ or send a written request to Case Program, John F. Kennedy School of Government, Harvard
University, 79 John F. Kennedy Street, Cambridge, MA 02138.
This document is authorized for use only in Uniandes Pregardo Introduccion Politicas Publicas (2016-I) SG by Sandra Garcia, Universidad de Los Andes - Colombia (UniAndes) from January
2016 to May 2016.
positions in several city agencies such as the Department of Housing. White’s career spanned both the public and
5
private sectors and along the way she had developed a reputation for being a “doer.” In the summer of 2006,
Mayor Bloomberg and Deputy Mayor for Health and Human Services, Linda Gibbs approached White to take on
the task of implementing the Commission’s recommendations. White had daunting challenges ahead. She had to
raise funds, redefine how poverty was measured, facilitate cross agency partnerships, and most important, devel-
op an effective and achievable evaluation system for all programs.
White described how the Center for Economic Opportunity (CEO) was first conceived. “We knew it
couldn’t just be me. So we decided on a center.” Mayor Bloomberg had made poverty reduction the top priority of
his second term. White had to hit the ground running and roll out initiatives at breakneck speed.
Mayor Mike
Michael Bloomberg was first elected as Mayor of New York City in 2001. He campaigned with promises to
turn around the city’s economy still reeling from the shock of the terrorist attacks in September 2001. One of the
richest men in the world, Bloomberg had amassed a fortune during an illustrious career on Wall Street and as
6
founder of the large media and information service company, Bloomberg LP. The entrepreneur turned politician
used his penchant for seeking profits in the business world into a focus on measurable results in government. “If
7
you can’t measure it, you can’t manage it,” was Bloomberg’s motto. When he was reelected by a landslide for a
second term in 2005, Bloomberg directed City Hall’s agenda toward fighting poverty. He created the Commission
for Economic Opportunity at a time when “no one else wanted to talk about the P word,” White said.
Big Apple, Small Gains in Addressing Poverty
New York City, the embodiment of the American dream had long held the promise of bigger, better fu-
tures for its residents. In 2006, however, one in five New Yorkers, or 1.5 million people, lived below the poverty
line. African Americans, Hispanics and households led by single females were disproportionately poor and often
unable to climb the economic ladder. Many of the poor were employed, but declining wages and high costs of
basic amenities like housing contributed to the problem. Even though the city offered a large infrastructure of wel-
8
fare programs, it faced an ongoing battle with poverty.
Innovation Fund
For the rest of 2006, White worked with Allegra Blackburn-Dwyer, the Commission’s only staff member,
and later, CEO’s Chief of Staff, to bring the Commission’s recommendations to life. CEO was located under the
5
Author interviews with Veronica M. White, Executive Director, Center for Economic Opportunity, April 9 and May 9, 2012.
6
“The World’s Billionaires: Michael Bloomberg,” Forbes.com, http://www.forbes.com/profile/michael-bloomberg/, accessed
May 24, 2012.
7
Roberts, S. “Calculating Poverty in New York: More by City Standard, and less by Federal One,” New York Times, March 2,
2010.
8
Poverty statistics from “Increasing Opportunity and Reducing Poverty in New York City,” Commission for Economic Opportuni-
ty Report to Mayor Bloomberg, September 2006; and examples of term-limited assistance include the Temporary Assistance for
Needy Families and Earned Income Tax Credit programs.
HKS Case Program 2 of 19 Case Number 1971.0
2016 to May 2016.
Mayor’s office with a broad mandate to facilitate working with other city agencies. Instead of providing direct ser-
vices, CEO would transfer funding directly to city agencies and local nonprofits. White modeled CEO as an innova-
tion lab that would test anti-poverty programs by applying a results-based approach. The Center began with an
annual fund of $100 million. Notably, Bloomberg and Gibbs demonstrated their commitment to CEO by allocating
9
$75 million of city funds to center programs every year. White raised the additional $25 million annually from sev-
eral charitable organizations such as the Rockefeller Foundation. In December 2006, Bloomberg announced the
creation of the Center for Economic Opportunity with White as the Executive Director. "The new Innovation Fund…
will carefully monitor new programs and hold them accountable for producing results - just as a business would.
10
And if we find a certain program isn’t making the grade, we will terminate its funding,” he stated.
Define Poor
Next came the more challenging task of setting in motion the Commission’s recommendations. White
needed a “two-pronged team”—one prong for measuring poverty in New York, and another for developing and
monitoring the new anti-poverty programs. Like the Commission members, Bloomberg was interested in gaining a
better understanding of poverty in New York City. The federal poverty measure was often criticized because it
counted only pre-tax income and ignored in-kind support such as housing subsidies or food stamps. “Mayor
Bloomberg comes from the data world,” said a fellow at the Manhattan Institute “if you don’t trust the data going
11
in, you can’t rely on the results coming out.” The Commission had suggested several strategies for measuring
poverty in New York, including creating an “Economic Opportunity Index” comprising a group of indicators that
reflected commonly held perceptions of poverty, or, conducting a longitudinal study that would explore the expe-
riences of those entering, exiting, or remaining in poverty. Instead of reinventing the wheel, however, White relied
on the knowledge of experts in the field—a technique she utilized often.
The United States National Academy of Sciences had created a new way to measure poverty in 1995. Left
largely untouched since, the alternative method was a radical departure from the federal poverty measure. Unlike
the widely used but increasingly inadequate official poverty measure, the Academy’s alternative redefined the
poverty threshold to reflect long-term changes in the standard of living and geographical differences in housing
12
costs. “[It] would have been a tremendous thing if we could have used the Academy’s work, tailored it to New
York City and created the first poverty measure based on that work,” White said. In 2007, White hired Mark
9
City funding for the Center for Economic Opportunity included $68 million in tax-levy dollars and state and federal funds total-
ing $7.7 million. Total CEO budget of $150 million also included $42 million for the Child Care Tax Credit, $11 million for educa-
tional programs for youth at risk and some private donations for the conditional cash transfer program.
10
Press Release December 18, 2006, “Mayor Bloomberg Announces $150 Million Annual Investment for Solutions to Challenges
Raised by the Commission for Economic Opportunity,”
http://www.nyc.gov/portal/site/nycgov/menuitem.c0935b9a57bb4ef3daf2f1c701c789a0/index.jsp?pageID=mayor_press_relea
se&catID=1194&doc_name=http%3A%2F%2Fwww.nyc.gov%2Fhtml%2Fom%2Fhtml%2F2006b%2Fpr437-
06.html&cc=unused1978&rc=1194&ndi=1, accessed May 14, 2012.
11
De Mause, Neil “Target of Opportunities” City Limits Investigates, Spring 2008, Vol. 32, No. 01.
12
The Academy also expanded the measure of resources available to families by including the effects of taxation, in-kind bene-
fits, and non-discretionary spending for commuting to work, childcare, and medical care. “Policy Affects Poverty: The CEO Pov-
erty Measure, 2005-2009,” Center for Economic Opportunity, March 2011.
2016 to May 2016.
Levitan to lead the poverty research team at CEO. Levitan, an economist by training, had conducted policy research
on poverty and joblessness and was deeply familiar with the Academy’s methodology. In fact, it was White’s inter-
est in adopting the model to New York that drew Levitan to CEO. “I don’t think I would have taken the job if that
13
hadn’t been the plan,” he said.
When the Academy’s methodology was applied to New York City, it raised the poverty threshold to nearly
$6,000 above the federal poverty line. The new, more comprehensive formula calculated the impact of taxes and
non-cash assistance (such as food stamps and housing subsidies) to pre-tax cash income. It also accounted for fam-
ily work-related expenditures (such as childcare) and out-of-pocket medical spending. The net result of the new
14
calculation increased the percentage of poor in New York from 18 to 23 percent.
Programs and Evaluations
White hired Kristin Morse to lead the CEO’s team on programs and evaluation. Like the majority of the
staff at the Center, Morse had not worked in government before. “I had the luxury of building a brand new unit. I
handpicked and hired everyone. It is very different from going in and taking over a unit where people are used to
doing things a certain way,” White said. Morse had done a broad range of anti-poverty work in New York City in-
cluding in housing and domestic violence.
Two distinct strategies guided CEO’s collaborations with the rest of city government. First, armed with the
leadership support of commissioners at the city agencies, CEO worked closely with deputy commissioners and sen-
ior program staff to roll out programs. Second, White and her staff made it clear from the outset that funding for
programs would be based on their performance. City agencies “understood that the programs were going to be
evaluated and they understood that certain data would need to be collected,” she said. White used another meas-
ure to increase ownership of the anti-poverty programs within city government. CEO transferred program funding
to city agencies to either deliver services directly or work through community-based organizations. The Center,
however, played the role of convener and met regularly with city agencies and their implementing partners.
With the mandate to work closely with city managers, and the responsibility to monitor CEO’s programs
regularly, White divided the $100 million fund. “We looked at what the Commission proposed and then looked at
what we could fund,” she said. “Clearly, we could have put all our money into one program but we wanted to
open, if we could, multiple sites for each program rather than just resting everything on one provider. If there was
an issue, we would never know if it was the program or the provider.” This focus on understanding what was work-
ing in the programs permeated CEO’s ethos. CEO shied away from setting aside a portion of each program’s budget
for evaluation. “There was always a tension between allocating enough money to deliver the program and leaving
15
enough to see if it was doing what it was supposed to do,” Morse explained. Instead, CEO set aside $2 million
13
Author interview with Mark Levitan, Director of Poverty Research, Center for Economic Opportunity, May 9, 2012.
14
Statistics from “The CEO Poverty Measure: A Working Paper,” Center for Economic Opportunity, August 2008, p. 3. The dif-
ference between the federal and CEO poverty rates was not uniformly distributed across the city’s population. The CEO poverty
rate was higher among elderly, non-white Hispanics, Asians and naturalized citizens and lower among children living in single
parent families.
15
Author interviews with Kristin Morse, Deputy Director, Center for Economic Opportunity, April 23 and May 9, 2012.
2016 to May 2016.
annually for evaluations of all programs funded by public finances. In the first year of operation, CEO launched 31
programs in partnership with more than 20 city agencies. The programs fell under three broad categories as rec-
ommended by the Commission: workforce development, youth development and supporting families with young
children.
The strategy to place evaluation at the core of CEO’s mission, however, was put to the test. “Early on we
found ourselves trying to explain to people that, of course, CEO could evaluate 40 different programs. But we met
with a lot of skepticism,” Morse said. CEO and city agencies met regularly to arrive at a common understanding of
each program’s goals. All city agencies were required to submit monthly reports or “real time narratives” of pro-
gram updates. In addition, each city agency had to file quarterly reports that provided data on progress toward
targets (such as number of people served etc., see Exhibit A for sample quarterly report). Morse also developed
partnerships with independent evaluation agencies to conduct rigorous evaluations on a per-program basis.
The initial slate of CEO programs varied widely both in scale and scope. Some programs were bold and
new to the United States like Family Rewards, and other programs had previously demonstrated positive impacts
like Jobs-Plus—but all conformed in letter or spirit to the Commission’s vision.
Program 1: Family Rewards
Perhaps the most controversial of CEO’s programs, Opportunity NYC-Family Rewards, was based on an
approach called conditional cash transfers (CCTs), popular in developing countries. Under these programs, gov-
ernments provided cash payments to the poor if they complied with conditions such as sending children to school
or taking them to the doctor. Even though the Commission had not recommended piloting a CCT program in New
York City, Mayor Bloomberg and several Commission members were interested in experimenting with the model.
“Historically, the rest of the world has looked to America for leadership in social policy, but there is no reason that
16
we cannot also learn from the experience of others,” said Bloomberg.
Mexico’s first CCT, Progresa, was created in 1994. The innovative program tried to motivate behavioral
change by conditioning payments on actions that could improve the lives of future generations. From its humble
start in remote villages, the immensely successful Progresa was rechristened Oportunidades and soon became the
Mexican government’s signature anti-poverty vehicle. As one of the most widely researched social programs in the
world, Oportunidades, stood up to the test of rigorous evaluations and showed numerous positive impacts, includ-
17
ing higher school enrollment rates and improved nutrition among children. The World Bank began promoting
CCTs as a powerful method to fight entrenched poverty, and other developing countries adopted the methodology
16
De Mause, Neil “Target of Opportunities” City Limits Investigates, Spring 2008, Vol. 32, No. 01.
17
“Conditional Cash Transfers: Reducing Present and Future Poverty,” World Bank Policy Research Report, February 2009, pp.
127-164.
2016 to May 2016.
18
with gusto. Brazil’s CCT program became the largest in the world, but as in other countries, the model demon-
19
strated only modest results in urban areas.
The proposal to develop a CCT program in New York met with significant controversy. Handing out condi-
tional cash payments was offensive to some liberals who believed the requirements embedded within the system
infantilized the recipients. “[Family Rewards] comes out of the idea that poor people are almost sort of culturally
and inherently dysfunctional. Not because of structural circumstances but because of their own personal failings,”
20
complained Mark Griffith at the Drum School of Public Policy in New York City. Some conservatives objected to
what they viewed as yet another welfare program that increased dependence on the state. Heather MacDonald at
the Manhattan Institute said “If [Family Rewards] goes large scale, it will further break down the moral obligation
21
to care for one’s child.” “I was asked often,” White explained “why are you paying people to send children to free
schools and to access free medical care? The next thing you know, you’ll pay poor people to brush their teeth.”
Despite the criticism, White, with the support of Bloomberg and Gibbs, planned to launch Family Rewards
in 2007. CEO bypassed the ideological uproar by using private resources for the program. Funders included the
22
Mayor’s own Bloomberg Philanthropies and the Rockefeller Foundation. Unlike other programs launched in CEO’s
first year, Family Rewards was not housed within a city agency. With a total budget of approximately $44 million,
the program would span three years and would be the first large-scale conditional cash transfer program in the
23
United States.
Once again, White tapped into the knowledge of experts. Family Rewards was led by a “core design team”
including White, Deputy Mayor Gibbs and James Riccio, veteran social policy evaluator at MDRC, a well known so-
cial science research organization. “CEO was not a very big organization and they didn’t have the internal capacity
to design and operate Family Rewards. I think they very quickly realized that they needed to find the right exper-
24
tise to join them in the effort,” Riccio explained. In early 2007, Bloomberg, White, Gibbs, Riccio and others visited
Mexico to learn first-hand about Oportunidades. The Rockefeller Foundation funded the design stage of Family
Rewards, including a learning exchange with the government of Mexico and researchers who evaluated
Oportunidades. “There was a body of rigorous evidence [on CCTs] we quickly learned about,” said Riccio,
18
“Anti Poverty Programmes: Give the Poor Money,” Economist Intelligence Unit, Executive Briefing, August 6, 2010.
19
“Brazil’s Bolsa Familia: How to Get Children out of Jobs and Into School,” The Economist, July 29, 2010.
20
Goldstein, D. “Behavioral Theory: Can Mayor Bloomberg Pay People to do the Right Thing?” The American Prospect, August
14, 2009.
21
Rosenberg, T. “A Payoff Out of Poverty,” New York Times, December 21, 2008.
22
Other funders included the Starr Foundation, Open Society Institute, Robin Hood Foundation, the Tiger Foundation, the An-
nie E. Casey Foundation, American International Group, the John D. and Catherine T. MacArthur Foundation, and the New York
Community Trust. Private funding went through the Mayor’s Fund to Advance New York City, not-for-profit organization, which
facilitated innovative public-private partnerships throughout NYC’s five boroughs.
23
Budget numbers for Family Rewards and other CEO programs in this case were obtained from CEO staff. Small scale cash
transfers like the Minnesota Family Investment Plan had been attempted in the U.S. before but not at the scale of Family Re-
wards.
24
Author interview with James Riccio, Director, Low Wage Workers and Communities Policies Area, MDRC, May 9, 2012.
2016 to May 2016.
“[Oportunidades] was based on randomly assigning villages, not individuals, to the program but that was at a vastly
larger scale than our experiment in New York.”
The root causes of poverty in New York, however, were different than those in Mexico. More important,
poor New Yorkers already had access to a large welfare system. Adapting the model was going to be a challenge.
“We wanted to make sure we had not only an academic perspective, but also an operative and a government per-
spective at the time of the design,” Riccio said. CEO solicited inputs from city agencies including the Departments
of Health and Education, local nonprofits, and several academics. CEO ultimately settled on 22 different incentives
that were divided into three categories: education-focused conditions, health-focused conditions and workforce-
focused conditions. Rewards ranged from $20 to $600 each for children performing well in school, for family
members getting preventive health and dental check-ups, and for parents working full time (see Exhibit B for full
schedule of incentives). Family Rewards incentives for school achievement and parents’ employment were new to
the CCT model and not part of Oportunidades or other CCTs. On average, families could earn approximately $3,000
in each year of the program. They received payments in the form of direct deposits into bank accounts or cash
cards every two months. Constraints of the budget limited the number of eligible families to 2,400— the majority
of which were led by single mothers.
“Family Rewards was targeted toward families who lived in selected community districts
and who had incomes at or below 130 percent of the federal poverty level. Eligible fami-
lies had to have at least one child in the fourth, seventh, or ninth grade. These grades
were selected because they [were] at or near the start of critical transition points in ed-
25
ucation.”
Family Rewards was implemented by a network of six local organizations, which were responsible for re-
cruiting and working with families in target neighborhoods. “Implementation challenges were there from the very
beginning,” said Riccio, “because we unfortunately did not have a pilot phase where we could try out the program
on a small scale before ramping it up.” Neighborhood organizations reported problems in recruiting the families
MDRC had identified for the program. “People thought it was a scam. They were surprised and couldn’t believe we
were going to give money for their children doing well in school,” Riccio said. He described another challenge in
explaining all the components on offer, “there were 22 different incentives and it was a difficult educational effort
to help families understand what they had to do and how they could earn rewards.”
Program 2: City University of New York-Accelerated Study in Associated Programs
Community colleges in the United States did not have the same aura as the nation’s renowned universi-
ties, but enrolled more than half of its 20 million undergraduates. Associate degree programs offered at communi-
ty colleges were a quick and affordable way to get a degree for many poor and first-generation immigrant stu-
26
dents. In theory, students could have graduated after two and a half years of full-time study at a community col-
25
Riccio, J. et al “Toward Reduced Poverty Across Generations: Early Findings from New York City’s Conditional Cash Transfer
Program,” MDRC, March 2010, p. ES-6.
26
“Restoration Drama: America’s Under-appreciated Community Colleges Hold Promise,” The Economist, April 28, 2012.
2016 to May 2016.
lege. High enrollment rates, however, had not manifested into high graduation rates. Across the country only 17
percent of those enrolled in urban community colleges graduated within six years. In New York City, the gradua-
27
tion rate was marginally higher at 21 percent. Community college students often had competing commitments,
including full or part-time jobs and childcare responsibilities that got in the way of attending classes. Even though
an associate’s degree commanded a significantly higher income than a high school diploma, community colleges
28
were failing to both graduate students and to align training with the needs of employers.
The City University of New York (CUNY), a large public university system, encompassed several research
universities and six community colleges. In late 2006, CUNY’s Chancellor, Matthew Goldstein, approached CEO
with a proposal—CUNY-Accelerated Study in Associate Programs (ASAP) for the city’s community colleges. The
proposal drew on a growing belief that community college students needed more support to earn their degrees in
a timely manner. CUNY-ASAP required low-income community college students to attend classes in consolidated
“block” course schedules, at the same time of day or on weekends, to accommodate their work and family re-
quirements. “The other important benefit of block scheduling was that students had the support of being with a
group of peers through many of their core courses,” Morse explained. Students were required to take 12 credits
each semester, enabling them to receive full time financial aid. They were also required to select a narrow set of
majors and to meet regularly with pre-assigned advisors and employment specialists, and, if identified as struggling
academically, to attend tutoring for extra support. Graduates were primed for jobs in high growth fields like health
and hospitality or to successfully transfer to 4-year colleges for baccalaureate study. “Essentially, it was a collection
of interventions designed to encourage students to get their associate’s degrees faster. The program had every-
thing from financial assistance to tuition waivers, from free books to free metro cards,” said Morse. CUNY-ASAP’s
ambitious goal was to assist 50 percent of an entering class of 1,000 students to graduate within three years.
White gave CUNY-ASAP the green light in January 2007. CUNY was going to operate the program for three
years with an annual budget of $6.5 million. The first cohort of 1,000 students needed to be enrolled by September
2007. “That’s a very fast start-up pace to create an entirely new program,” said Donna Linderman, Director of
29
CUNY-ASAP. “Recruitment was always a challenge. Community college students tended to do things on a much
more leisurely time frame. They applied and waited to be tested. They tested, and then they waited to apply for
financial aid. In our case, we really needed them to take all of these steps as quickly as possible,” she said.
CUNY’s Office of Institutional Research and Assessment collected robust administrative data on students.
There was rich detail on student attendance and course completion but CUNY could not share students’ personal
information with external partners. “We were running into issues of confidentiality and data sharing,” Morse ex-
plained.
27
Center for Economic Opportunity: CUNY-ASAP, http://www.nyc.gov/html/ceo/html/programs/cuny_asap.shtml, accessed
May 17, 2012.
28 st
“Reclaiming the American Dream: Community Colleges and the Nation’s Future,” Report from the 21 Century Commission
on the Future of Community Colleges,” 2012, pp. 9-14.
29
Author interview with Donna Linderman, Director of CUNY-ASAP, May 8, 2012.
2016 to May 2016.
Program 3: Teen ACTION
Drugs and sex served as a gateway to pregnancies, substance abuse and school suspensions for teenagers
living in poverty in New York City. Such high-risk behaviors set the stage for a downward spiral at a critical point in
the lives of these teenagers, and perpetuated the cycle of poverty. New York City had seen a steady decline in teen
births between 1990 and 2000, but the teenagers who continued to get pregnant were overwhelmingly poor and
30
unmarried.
Teen ACTION (Achieving Change Together in Our Neighborhood) was developed by CEO and the New York
City Department of Youth and Community Development (DYCD) in response to this challenge. It was an after
school service learning program with the overarching goal of reducing risky behaviors, promoting youth develop-
ment and improving community engagement among teenagers in high poverty neighborhoods. Teen ACTION’s
curriculum tried to redirect the attention of students to gaining skills they could use in the future. It brought to-
gether several service learning elements including structured classroom activities, community service, group re-
flections and opportunities for participants to engage in decision-making.
Carson Hicks was the deputy Director of Programs and Evaluation at CEO and managed the youth devel-
opment portfolio. Hicks, a sociologist by training, worked closely on the development of Teen ACTION. The pro-
gram design was a variation on the theme of several successful service learning initiatives including the Teen Out-
reach Program—developed in 1978 and implemented in cities across the United States. The Teen Outreach Pro-
gram was rigorously evaluated and had proven to significantly reduce rates of teen pregnancy, course failure and
31
school suspensions among participants when compared to a control group. CEO also found “studies of various
types of after school programs to have consistently shown that the longer participants remained in the program,
32
the greater its impact.”
In the classroom component of the Teen ACTION curriculum, students were exposed to topics such as
immigration, the environment and sexual and reproductive health, while also being taught how to work in teams
and effectively solve problems. As part of the service learning component of the curriculum, students were re-
quired to research and identify problems in their communities and design projects to serve residents in need. Ef-
forts included helping senior citizens, tutoring younger students and conducting research on nutrition or sex edu-
cation. “The curriculum taught a host of issues like recycling and global warming,” explained Hicks. “After learning
about these issues, the students looked around in their neighborhoods to see if there was an activity or project
33
that they could undertake to help improve their communities.”
30
Center for Economic Opportunity: Teen ACTION, http://www.nyc.gov/html/ceo/html/programs/teen_action.shtml, accessed
June 18, 2012.
31
Allen, J. et al “Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Ap-
proach,” Child Development, August 1997, Vol. 64, pp. 729-742.
32
Center for Economic Opportunity: Teen ACTION, http://www.nyc.gov/html/ceo/html/programs/teen_action.shtml, accessed
June 18, 2012.
33
Author interviews with Carson Hicks, Director of Programs and Evaluation, Center for Economic Opportunity, April 23 and
May 9, 2012.
2016 to May 2016.
Between fall 2007 and spring 2008, with a budget of $3.8 million, Teen ACTION was deployed in 60 after
school sites via DYCD. The program served more than 3,000 young adults between the ages of 13 and 21 years,
enrolled in grades 6 to 12. Hicks was aware of the bureaucratic red tape involved in getting permissions to talk to
students in schools. “We had to get approval from the Department of Education. We also needed to get consent
from students and their parents because the participants were minors,” she explained. There were challenges in
getting Institutional Review Board approvals as well.
Program 4: Jobs-Plus
During one of the intense collaborations for Family Rewards, White and Riccio had a conversation on an
entirely different program called Jobs-Plus. “Given that [White’s] mission was to promote evidence-based policy,
and that she also wanted to do more to address the needs of public housing residents, I said to her, ‘let me tell you
about Jobs-Plus because I think it’s relevant to what you do at CEO,’” recalled Riccio. Jobs-Plus was a “saturation”
program that targeted all residents in a public housing development. Residents received employment and training
34
services and incentives designed to “make work pay.” Residents also played an important role in sharing infor-
mation about work related opportunities with their neighbors.
Between 1998 and 2003, MDRC, with support from several federal agencies and private foundations, in-
cluding the U.S. Department of Housing and Urban Development and the Rockefeller Foundation, conducted a
35
“large-scale demonstration” of Jobs-Plus in six cities. Jobs-Plus had a built-in evaluation strategy. Several public
housing buildings picked randomly in each city received Jobs-Plus services, and other public housing buildings did
not—automatically serving as a comparison group. The impact of the program was determined by comparing the
long-term earnings trends (before and after the program began) of residents who lived in the housing develop-
ments that offered Jobs-Plus with those of residents of comparison developments. When implemented as pre-
scribed, Jobs-Plus increased public housing residents’ income by 16 percent per year (relative to the comparison
group) for the seven-year follow-up period. Overall, Jobs-Plus helped some participants get new jobs and others to
36
maintain and improve on existing jobs. “Here was a great program with a big fat evaluation just sitting up on a
shelf and nobody was paying attention to it,” White said. “We had to dust it off.” White was also excited because
“if somebody else paid for the evaluation, all the better, given the cost of evaluations.” CEO worked with the New
York City Housing Authority for Jobs-Plus and invited the City University of New York and the Human Resources
Administration to join in the implementation.
Jobs-Plus had to be adapted to unique circumstances in New York. “One public housing development in
New York City could have fit all of the Jobs-Plus housing developments that were in our study. The scale was mas-
sively different,” Riccio said. With an initial budget of $1 million, CEO wanted to start with one site. “We essentially
34
Jobs-Plus was not recommended by the Commission for Economic Opportunity but was an example of a program that could
make work pay—a consistent theme in the Commission’s report.
35
The six cities were Baltimore, Chattanooga, Dayton, Los Angeles, St. Paul and Seattle.
36
Center for Economic Opportunity: Jobs-Plus, http://www.nyc.gov/html/ceo/html/programs/jobs-plus.shtml, accessed May
21, 2012.
2016 to May 2016.
looked for a development that had approximately a thousand families, and a significant number of working age
residents,” explained Morse. CEO began by testing the program with one housing complex in Manhattan.
Program 5: Employment Works
New York City had a complex jail system that incarcerated approximately 100,000 individuals every year.
Former criminals often had difficulty finding and holding down jobs and were in many cases rearrested or re-
mained unemployed for long periods of time. Every year nearly 30,000 people passed through the revolving doors
37
of the New York City Department of Probation. Typically, half of the probationers were unemployed. In other
parts of the U.S., comprehensive employment services for individuals on parole and probation showed significant
38
decreases in recidivism. CEO developed Employment Works to provide education, training and support services
to probationers and prepare them for sustained, long-term employment. Morse fostered partnerships with the
Department of Probation and the Department of Small Business Services to reach approximately 600 probationers
in the first year of the program. The goal was to help each unemployed or underemployed probationer find and
maintain a job that paid at least $9 an hour, for at least one year. The budget for Employment Works was approx-
imately $1.75 million.
Once Employment Works was launched, Morse undertook efforts to work with administrative data col-
lected by the Department of Probation. A critical piece of information however had never been collected. “We
expected to have good information from the Department of Probation on whether people referred into the pro-
gram were employed or not, kind of a basic thing,” she said. “We knew a ton of information about the people who
were served. We knew their education and we knew how serious their charges were. We knew whether or not
they ended up getting a job but we didn’t know their employment history,” Morse explained. Some of the infor-
mation might have been available in the recipient case notes but that would mean spending more time and money
trolling through imperfect data. “What do you do when six months into a program you realize the agency’s admin-
istrative record did not collect this super obvious piece of information?” Morse asked.
Evaluation Strategies
Several evaluation methods were available to organizations and researchers as they tried to measure so-
cial programs:
Types of Evidence Description
Program Design Evaluations Assessed how well the program theory met the demands of the target population.
or Formative Assessments For example, logic model/framework approaches or theories of change.
Monitoring Ongoing collection of data on the status of program implementation. Information
37
Statistics from Center for Economic Opportunity: Employment Works,
http://www.nyc.gov/html/ceo/downloads/pdf/appendixb_employmentworks.pdf, accessed May 21, 2012.
38
“The Impact of Employment on Violations file on Federal Offenders in Eastern Missouri 1999-2004,” United States District
Court Eastern Missouri Probation Office.
2016 to May 2016.
collected could include share of the target population who received the service;
how much of the funds were used etc.
Process Evaluations Analyzed effectiveness of program implementation and delivery. Assessment de-
termined, among other things, if the organization was adequately staffed and
trained to deliver services to the target population, if the funds were being used
39
effectively etc.
Impact Evaluations Determined overall effects of the program. Impact evaluations measured the im-
pact of the program based on outcomes achieved.
For CEO though, evidence based programming was replete with obstacles. Each program had its own
share of complications, either inherent or unexpected. White and her team could use the following evaluation
40
tools, with varying associated costs, to gather data on any or all types of evidence listed above.
Evaluation Tool Description Cost
Survey CEO in collaboration with city agencies and external Cost was based on number of peo-
evaluators could design a set of questions to assess the ple to be surveyed and number of
impact of the program. Reliability of any survey depend- questions. On average, a survey
ed on the “response rate,” or the percentage of recipi- would cost $300,000 - $500,000 for
ents who answered the survey. 1,000 individuals. Costs were higher
for minors because researchers
would need to get the permission of
parents or guardians in order to
interview youth.
Focus Group CEO could commission a focus group of 10-20 partici- Cost ranged between $10,000 and
pants in a program with a facilitator. Questions would $20,000.
be predetermined by the city agency and/or the exter-
nal evaluator. Reliability of focus group responses de-
pended on the skill of the facilitator.
Random The “gold standard” in evaluation. CEO could determine Minimum cost for random assign-
Assignment random assignment of treatment and control groups in ment was $1 million but could run
partnership with external evaluators and city agencies. up to $5 million. Cost of random
assignment studies were driven
39
Rossi, P. et al, “Evaluation. A Systematic Approach,” Thousand Oaks: Sage Publications, 1999.
40
Costs based on interviews with Carson Hicks, Kristin Morse and James Riccio.
2016 to May 2016.
heavily by data collection costs.
CEO’s financial challenges increased in 2008, when the economic recession led to cuts in the city budget.
Tough decisions needed to be made. CEO had to terminate programs that did not perform well and had to reduce
its evaluation budget from $2 million to $1.2 million. With constant pressure “to produce results very quickly,” the
team had to make the most of limited resources while judiciously measuring program performance. What could
CEO do to gather evidence on its programs?
2016 to May 2016.
Exhibit A
Sample Center for Economic Opportunity Quarterly Report
YEARLY
DEMOGRAPHIC INDICATOR INDICATOR FISCAL FISCAL TARGET
PROGRAM NAME AGENCY INDICATOR TYPE CATEGORY INDICATOR NAME ID VALUE YEAR QUARTERVALUE
Employment Works SBS Program Process N/A Number of New Entrants 80103001 268 2010 3 1,454
Employment Works SBS Program Process N/A Number of Customers Enrolled in Skills Development 80103002 257 2010 3
Employment Works SBS Program Process N/A Number of Customers Completed Skills Development 80103003 150 2010 3
Employment Works SBS Program Process N/A Average TABE Score at Program entry 80103004 7.39 2010 3
Employment Works SBS Program Process N/A Number Enrolled in Basic Education 80103005 2 2010 3
Employment Works SBS Program Process N/A Number Enrolled in GED 80103006 14 2010 3
Employment Works SBS Program Process N/A Number Enrolled in Vocational Training 80103008 91 2010 3
Employment Works SBS Program Process N/A Number Completed in Vocational Training 80103009 91 2010 3
Employment Works SBS Program Process N/A Number Screened for Work Supports 80103010 102 2010 3
Employment Works SBS Participate OutcomeN/A Number of Job Placements 80102001 158 2010 3 762
Employment Works SBS Participate OutcomeN/A Number of Job Placements at $9 per Hour or More 80102002 88 2010 3 437
Employment Works SBS Participate OutcomeN/A Percent of Job Placements at $9 per Hour or More 80102003 56% 2010 3
Employment Works SBS Participate OutcomeN/A Median Hourly Wage at Placements 80102004 9 2010 3
Employment Works SBS Retention Outcome N/A # of Participants Eligible for Six-Month Rentention 80104001 124 2010 3
Employment Works SBS Retention Outcome N/A # of Participants Eligible for Twelve-Month Rentention 80104002 160 2010 3
Employment Works SBS Retention Outcome N/A # of Participants Confirmed Reaching Six-Month Retention 80104003 81 2011 2 404
Employment Works SBS Retention Outcome N/A # of Participants Confirmed Rteaching Twelve-Month Retention80104004 71 2011 2 159
Employment Works SBS Retention Outcome N/A Percent of Participants Confirmed Reaching Six-Month Retention 80104005 65% 2011 2
Employment Works SBS Retention Outcome N/A Percent of Confirmed Reaching Twelve-Month Retention 80104006 44% 2011 2
Employment Works SBS Demographic Age 18 to 20 80101002 58 2011 2
Employment Works SBS Demographic Age 65+ 80101006 0 2011 2
Employment Works SBS Demographic Age Unreported 80101007 0 2011 2
Employment Works SBS Demographic Gender Male 80101008 234 2011 2
Employment Works SBS Demographic Gender Female 80101009 34 2011 2
Employment Works SBS Demographic Gender Other 80101010 0 2011 2
Employment Works SBS Demographic Gender Unreported 80101011 0 2011 2
Employment Works SBS Demographic Race Asian/Pacific Islander 80101012 7 2011 2
Employment Works SBS Demographic Race Black (non-Hispanic) 80101013 147 2011 2
Employment Works SBS Demographic Race White 80101014 21 2011 2
Employment Works SBS Demographic Race Hispanic 80101015 76 2011 2
Employment Works SBS Demographic Race Other 80101016 4 2011 2
Employment Works SBS Demographic Race Unreported 80101017 13 2011 2
Employment Works SBS Demographic Dependents 0 80101018 2 2011 2
Employment Works SBS Demographic Dependents 3 or more 80101021 2 2011 2
Employment Works SBS Demographic Dependents Unreported 80101022 263 2011 2
Employment Works SBS Demographic Borough Bronx 80101023 34 2011 2
Employment Works SBS Demographic Borough Brooklyn 80101024 88 2011 2
Employment Works SBS Demographic Borough Manhattan 80101025 24 2011 2
Employment Works SBS Demographic Borough Queens 80101026 100 2011 2
Employment Works SBS Demographic Borough Staten Island 80101027 20 2011 2
Employment Works SBS Demographic Borough Other 80101028 2 2011 2
Employment Works SBS Demographic Borough Unreported 80101029 0 2011 2
Employment Works SBS Demographic Education Level Some High School 80101031 114 2011 2
Employment Works SBS Demographic Education Level High School Diploma 80101032 60 2011 2
Employment Works SBS Demographic Education Level GED 80101033 41 2011 2
Employment Works SBS Demographic Education Level Some College/Vocational (No Degree Completed) 80101034 40 2011 2
Employment Works SBS Demographic Education Level Vocational 80101035 1 2011 2
Employment Works SBS Demographic Education Level Associates 80101036 5 2011 2
Employment Works SBS Demographic Education Level Bachelors 80101037 7 2011 2
Employment Works SBS Demographic Education Level Post Graduate Degree 80101038 0 2011 2
Employment Works SBS Demographic Education Level Unreported 80101039 0 2011 2
Source: Adapted from quarterly report on Employment Works, Center for Economic Opportunity.
2016 to May 2016.
Exhibit B
The Opportunity NYC Demonstration: Family Rewards Schedule of Rewards

_____________________________________________________________________________________________
Activity Reward Amount
_____________________________________________________________________________________________
Education incentives
Elementary and middle school students
Attends 95% of scheduled school days (discontinued after Year 2) $25 per month
Scores at proficiency level (or improves) on annual
math and English language arts (ELA) tests
Elementary school students $300 per math test; $300 per ELA test
Middle school students $350 per math test; $350 per ELA test
Parent reviews low-stakes interim tests (discontinued after Year 1) $25 for parents to download, print, and
review results (up to 5 times per year)
Parent discusses annual math and ELA test results with $25 (up to 2 tests per year)
teachers (discontinued after Year 2)
High school students

Attends 95% of scheduled school days $50 per month
Accumulates 11 course credits per year $600
Passes Regents exams $600 per exam passed (up to 5 exams)
Takes PSAT test $50 for taking the test (up to 2 times)
Graduates from high school $400 bonus
All grades
Parent attends parent-teacher conferences $25 per conference (up to 2 times per year)
Child obtains library card (discontinued after Year 2) $50 once during program
Health incentives
Maintaining public or private health insurance (discontinued after Year 2)
For each parent covered Per month: $20 (public); $50 (private)
If all children are covered Per month: $20 (public); $50 (private)
Annual medical checkup $200 per family member (once per year)
Doctor-recommended follow-up visit (discontinued after Year 2) $100 per family member (once per year)
Early-intervention evaluation for child under 30 months old, $200 per child (once per year)
if advised by pediatrician
Preventive dental care (cleaning/checkup) $100 per family member (once per year for
children 1-5 years old; twice per year for family
members 6 years of age or older)
Workforce incentives
Sustained full-time employment $150 per month
Education and training while employed at least 10 hours per week Amount varies by length of course, up to a
maximum of $3,000 over three years
Source: “Toward Reduced Poverty across Generations: Early Findings from New York City’s Conditional Cash Trans-
fer Program,” MDRC, March 2010, p. ES-4.
2016 to May 2016.
Exhibit C
 Evaluation Firms
Source: Adapted from Center for Economic Opportunity presentation.
2016 to May 2016.
Exhibit D
Source: Center for Economic Opportunity Annual Report 2012, p. Executive Summary 7.
2016 to May 2016.
Exhibit E
2016 to May 2016.
Exhibit E (continued)
Source: Center for Economic Opportunity Annual Report 2012, pp. Executive Summary 8-9.
2016 to May 2016.

New York City Center For Economic Opportunity: An Evidence-Based Approach To Alleviate Poverty

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

New York City Center For Economic Opportunity: An Evidence-Based Approach To Alleviate Poverty

Uploaded by

Copyright:

Available Formats

HKS724

Case Number 1971.0

New York City Center for Economic Opportunity: An Evidence-Based

Certitude is not the test of certainty.

Big Apple, Small Gains in Addressing Poverty

HKS Case Program 2 of 19 Case Number 1971.0

HKS Case Program 3 of 19 Case Number 1971.0

Programs and Evaluations

HKS Case Program 4 of 19 Case Number 1971.0

Program 1: Family Rewards

HKS Case Program 5 of 19 Case Number 1971.0

HKS Case Program 6 of 19 Case Number 1971.0

Program 2: City University of New York-Accelerated Study in Associated Programs

HKS Case Program 7 of 19 Case Number 1971.0

HKS Case Program 8 of 19 Case Number 1971.0

HKS Case Program 9 of 19 Case Number 1971.0

HKS Case Program 10 of 19 Case Number 1971.0

Program 5: Employment Works

Types of Evidence Description

Monitoring Ongoing collection of data on the status of program implementation. Information

HKS Case Program 11 of 19 Case Number 1971.0

Evaluation Tool Description Cost

HKS Case Program 12 of 19 Case Number 1971.0

HKS Case Program 13 of 19 Case Number 1971.0

HKS Case Program 14 of 19 Case Number 1971.0

The Opportunity NYC Demonstration: Family Rewards Schedule of Rewards

High school students

HKS Case Program 15 of 19 Case Number 1971.0

Source: Adapted from Center for Economic Opportunity presentation.

HKS Case Program 16 of 19 Case Number 1971.0

HKS Case Program 17 of 19 Case Number 1971.0

HKS Case Program 18 of 19 Case Number 1971.0

HKS Case Program 19 of 19 Case Number 1971.0

You might also like