See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/24420610

After-school programs for adolescents: A review of evaluation research

Article  in  Adolescence · February 2009

Source: PubMed

CITATIONS READS
41 6,807

1 author:

Robert Apsler
Harvard Medical School
27 PUBLICATIONS   477 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Robert Apsler on 02 March 2014.

The user has requested enhancement of the downloaded file.

 After-School Programs 1

Running Head: AFTER-SCHOOL PROGRAMS: A REVIEW

This study was supported, in part, by funding from the United Way of Rhode Island. The

author gratefully acknowledges Scott Formica and Jacquelin Apsler for their insightful

comments on earlier drafts of this paper. Opinions expressed in this document are those of the

author and do not necessarily represent the official position or policies of the funder.

Robert Apsler, Department of Psychiatry, Harvard Medical School; Social Science

Research & Evaluation, Inc.

Requests for reprints should be sent to Robert Apsler, Social Science Research &

Evaluation, Inc., 84 Mill Street, Lincoln, MA 01773. E-mail: rapsler@ssre.org

AFTER-SCHOOL PROGRAMS FOR ADOLESCENTS:

A REVIEW OF EVALUATION RESEARCH

Robert Apsler
 After-School Programs 2

Abstract

During the last ten years, an infusion of private foundation and government funding markedly

increased the number of after-school programs targeting adolescents. This review focuses on the

quality of after-school program evaluation research. Numerous evaluations of after-school

programs exist, but serious methodological flaws limit the conclusions that can be drawn with

confidence from most of the studies. Major obstacles to conducting sound evaluations include

difficulties in obtaining appropriate comparison groups and dealing with sporadic attendance and

attrition. The review summarizes promising results, discusses the extent to which after-school

programs have achieved their goals, describes characteristics associated with successful after-

school programs, and reports on efforts to assess the cost effectiveness of after-school programs.
 After-School Programs 3

After-School Programs for Adolescents: A Review of Evaluation Research

This literature review describes the goals of after-school programs and examines the

degree to which programs have achieved those goals. Early enthusiasm for after-school programs

led to rapid growth in their numbers but has not been accompanied by research sufficiently

rigorous to produce unambiguous conclusions. Although numerous studies have been conducted,

few satisfy the criteria necessary to be considered methodologically sound. While this review

includes studies of varying quality, the impact of each study’s findings is weighted in proportion

to the quality of the research.

The Impetus For After-School Programs

Several factors energized the after-school movement. Community pressure for utilizing

school buildings following the end of the school day led Congress in 1994 to fund the 21st

Century Community Learning Centers (Dynarski et al., 2003). Parents demanded help caring for

their children during the period between the end of school and the time when parents arrived

home from work (Kane, 2004). These demands grew as increasing rates of caregivers entered the

workforce, and large numbers of youth were left without adult supervision during the after-

school hours. The National Institute on Out-of-School Time (2003) estimated that approximately

eight million children between the ages of 5 and 14 were often unsupervised after school in

1999. Estimates from this time period suggest that over two-thirds of school-age children did not

have parental supervision after school (Bureau of Labor Statistics, 2000; Long & Clark, 1997).

The specific type of care requested by many parents reflected growing emphasis on academic

performance and accountability, due in part to the No Child Left Behind Act of 2001 (Dynarski,

et al., 2004; Lauer et al., 2004).

 After-School Programs 4

Interest in after-school programs increased markedly following reports that juvenile

crime peaked between 2 p.m. and 6 p.m. on school days (Snyder, Sickmund, & Poe-Yamagata,

1996; Gottfredson, Gottfredson, and Weisman, 2001). Research documented an association

between the presence of parental supervision and lower levels of delinquent behavior, substance

use, and high-risk sexual behavior (Biglan et al., 1990; Block, Block, & Keyes, 1988;

Cernkovich & Giordano, 1987; Dishion, Patterson, Stoolmiller, & Skinner, 1991). Other research

demonstrated an association between lack of adult supervision and an increased likelihood of

risk-taking behaviors, victimization, and poor academic performance (Chung, 2000; Dwyer et

al., 1990; Newman, Fox, Flynn, & Christeson, 2000; Osofsky, 1999; Posner & Vandell, 1999;

Richardson et al., 1989).

Consequently, both private foundation and government funds increased to expand the

number of after-school programs. Funding for the 21st Century Community Learning Centers

jumped from $40 million in 1998 to roughly $1 billion in 2002 (Dynarski et al., 2004). The rapid

growth of after-school programming resulted from lobbying and grass roots efforts and was not

based on strong empirical findings (Fagan, 2007; Zief, Lauver, & Maynard, 2004).

Types Of After-School Programs

Reviewers of the literature on after-school programs have adopted differing criteria as to

what actually constitutes an after-school program. To date, no consensus exists in the field, and

no formal typological scheme grounded in theory has emerged. Investigators have categorized

after-school programs by program structure, content areas, and goals. For example, Hofferth,

Brayfield, Deich, and Holocomb (1991) grouped after-school programs based on six goals that

the programs were trying to achieve: (1) providing adult supervision and safe environments; (2)

providing a flexible, relaxed, and homelike environment; (3) providing cultural or enrichment
 After-School Programs 5

opportunities; (4) improving academic skills; (5) preventing behavior problems; and (6)

providing recreational activities. Fashola (1998) surveyed after-school programs and grouped

them according to five content-based categories: (1) language arts, (2) study skills, (3) academic

programs in other curriculum areas, (4) tutoring for reading, and (5) community-based programs.

Scott-Little, Hamann, and Jurs (2002) counted a program as an after-school program if it

operated during the hours after school dismissal and provided supervision for participants. The

authors found that some after-school programs focused on a single area, such as academics, while

others presented comprehensive experiences for participants. The individuals supervising students

could be regular schoolteachers who received extra pay or individuals with no other connection to

the school. Some programs incorporated one-to-one mentoring experiences, while most relied

entirely on much larger student-to-staff ratios. Many after-school programs operated in the

buildings where participants attended regular school because of transportation issues. Other

programs, such as the YMCA and Boys and Girls Clubs typically had their own facilities.

Riggs and Greenberg (2004) distinguished between two types of after-school programs.

One type simply provided a safe environment. Often unstructured, these programs could include

many types of recreational activities, homework time, arts and crafts, watching television, and

“hanging out.” Other programs purposefully sought to promote positive youth development.

These programs tended to be structured and often incorporated activities oriented toward

academic success. Some also emphasized social-emotional competence, computer skills, the arts,

and physical fitness.

 After-School Programs 6

METHODS

A review of the scientific literature was conducted in the latter part of 2007 to identify

evaluation studies of after-school programs. The review began with a formal search for articles

that had been indexed in the social science literature using Academic Search Premier (EBSCO),

PsycINFO, PsycARTICLES, and Educational Resources Information Center (ERIC). The initial

search term “after-school program evaluation” was broadened to “after-school programs” to

identify articles not specifically referred to as evaluation studies. A secondary search using

popular search engines on the World Wide Web revealed a number of other articles and

materials from websites such as the William T. Grant Foundation, Harvard Family Research

Project, Public/Private Ventures, American Youth Policy Forum, Chapin Hall Center for

Children at the University of Chicago, National Institute on Out-of-School Time, and the

Collaborative for Academic, Social, and Emotional Learning (CASEL).

Materials obtained in the search ranged from rigorous evaluations employing random

assignment to reports of after-school program staff’s subjective impressions of their programs’

operations. Some reports examined a single after-school program, while others reviewed large

numbers of programs. The reviews varied greatly in the care with which they assessed the

scientific rigor of the research and in the types of after-school programs covered. Findings from the

literature search validated the earlier observation by Scott-Little, Hamann, and Jurs (2002) that the

paucity of rigorous after-school program evaluations precluded conducting a meta-analysis.

RESULTS

The Results section begins with a discussion of limitations found in almost all the

reviewed materials. Next, studies assessing the overall impact of after-school programs are

organized roughly by the strength of their research designs. This is followed by research linking
 After-School Programs 7

outcomes to program characteristics. The review then addresses the question of whether after-

school programs reduce the number of unsupervised high-risk children, and it ends with a brief

review of efforts to examine the cost of after-school programs.

Limitations of After-School Program Research

Many reports of positive after-school program effects exist in the literature, yet nearly all

the research that generated these outcomes suffers from serious methodological flaws. The

dearth of sound evaluation research stymied some earlier reviewers’ efforts to summarize the

results of existing evaluations. For example, in an early study, Fashola (1998) reviewed 34

extended-day and after-school programs (some occurred entirely within the school day) focusing

on educational goals. The author found promising programs but qualified the endorsement due to

the absence of rigorous research designs. Despite some researchers’ use of quasi-experimental

designs, Fashola criticized virtually all the research for failing to avoid the possibility of

selection bias.

Similarly, Scott-Little, Hamann, and Jurs (2002) initially planned on conducting a meta-

analysis to provide an overview of the impact of after-school programs. However, they found too

few high quality evaluations to justify conducting a meta-analysis. The authors concluded that

while after-school programs may have positive impacts on children, more rigorous evaluations

are necessary to convincingly document this potential. In addition, the authors speculated that

after-school programs may have different effects on different subgroups of children – an

important possibility that tends to be ignored in research on after-school programs.

Finally, McComb and Scott-Little (2003) found 75 articles, reports, conference

presentations, and dissertations disseminated prior to spring 2002 reporting evaluations of after-

school programs. In their opinion, only two employed sound experimental designs. The authors
 After-School Programs 8

concluded that the results were mixed and difficult to summarize because of great variability in

results.

The most serious limitations of most after-school program evaluations include selection

bias, methods for tracking dosage, and program attrition.

Selection Bias

Two forms of selection bias exist in the majority of reviewed evaluations – selection bias

inherent in the research design and selection bias resulting from level of participation and

attrition.

Selection Bias in the Research Design. Selection bias in many studies stemmed from the

voluntary nature of participation in after-school programming. Since parents had to give

permission for their child(ren) to participate in an after-school program, differences may have

existed between children of parents who elected to give permission versus those of parents who

did not. There may also have been differences among youth wanting to participate in after-school

programs as compared to those who did not. Gottfredson, Cross, and Soulé (2007) suggested that

students motivated to join after-school programs “…are also those who are already on track for

prosocial development” (p. 290). Furthermore, these differences may manifest themselves in

better academic performance and more prosocial behavior. A more subtle form of selection bias

occurred in studies that compared participants in one school offering an after-school program

with students in another school where no program existed. Students in different schools may

have differed in ways that had the potential to affect their performance on measures of program

effectiveness. Even if students in two schools were, on average, identical, the fact that one school

introduced an after-school program and the other did not could reflect differences in the school

environments that affected all students.

 After-School Programs 9

Some researchers attempted to statistically control for socio-economic, ethnic, sex, and

other differences between students attending different schools. However, Campbell and Kenny

(1999) stated that “…statistical equating creates the illusion and not the reality of equivalence”

(p. 84). Thus, superior performance by after-school students in one school could be due to either

the types of students electing to participate and/or differences between students attending schools

with and without an after-school program.

Selection Bias by Level of Participation and Attrition. Another form of selection bias present in

many studies arose from attendance policies that placed no requirements on frequency of

participation. Sporadic attendance and high attrition were identified by the authors in many of

the studies reviewed. Consequently, analyses of after-school programs typically included only

students who chose to attend the programs frequently. The presence of low levels of attendance

and high attrition resulted in a double dose of selection bias. First, a select group of students

chose to enroll in an after-school program, and second, a select subgroup of these enrollees chose

to participate frequently in the program. Yet many studies compared the performance of this self-

selected subgroup of frequent participants with that of other students who chose not to enroll in

an after-school program. When investigators reported superior performance by after-school

participants, they generally failed to explain that the observed differences could have been due to

the characteristics of the students who selected themselves into the two groups and may have had

little or nothing to do with after-school program participation.

Methods for Tracking Dosage

Assessing the impact of after-school programs depends on knowing which students

enrolled and how frequently each one participated. Yet the complexity of what it means to

participate in a program was not addressed by many investigators. Most evaluations of after-
 After-School Programs 10

school programs simply counted the number of days on which students spent any time in the

program. Students who arrived for attendance but departed immediately afterwards and others

who remained without becoming involved have been counted as participating. But the after-

school program had had little opportunity to influence these youth who were included in

measures of participant performance. Thus, this approach to counting participants may have

underestimated the potential impact of an after-school program.

Dealing with attendance involves critical tradeoffs and is a major challenge for after-

school programs – not just for evaluators. On the one hand, after-school programs can assure

high attendance by accepting only those students willing to commit themselves to frequent

participation in the program. But there is a serious downside to such restrictive recruitment.

Grossman et al. (2002) identified programs that achieved high attendance by requiring students

to register for five days each week and compared them with other programs having a more

flexible enrollment policy. Programs with strict attendance requirements tended to have lower

enrollments than others with more flexible enrollment policies.

Low attendance figures were common for after-school programs. For example, Kane’s

(2004) evaluation of four 21st Century Community Learning Centers found that elementary

students participated in the program 1.9 days each week, on average, while middle school

student attendance averaged 0.9 days each week (Programs typically operated 4 or 5 days each

week.). Similar rates of participation have been reported for other after-school programs

(Grossman, et al., 2002; Walker & Arbreton, 2004).

Program Attrition

When participants drop out of a program, two immediate questions arise: Why did the

participants leave, and are there important differences between students who left and those who
 After-School Programs 11

remained? Weisman and Gottfredson (2001) sought answers to these questions in a study of 234

students enrolled in eight Maryland after-school programs. The authors interviewed 82% of the

77 participants who dropped out of the programs (33% dropout rate) and compared dropouts

with those students who remained. Dropouts scored significantly more at-risk on 11 out of 12

indicators than did students who remained in the program for a full year. For example, dropouts

had less parental supervision at home than students who remained in the program. Thus, students

with the greatest need for the programs’ help were least likely to take full advantage of the

intervention. A third of the dropouts reported dropping out because of boredom (the most

common reason offered). Another third of the dropouts left either because they moved away or

could not obtain transportation.

Studies of the Overall Impact of After-School Programs

Among the studies reviewed, Martin and colleagues (2007) reported some of the most

impressive behavioral changes among program participants. The study assessed 33 youth

attending an after-school program in an alternative school setting. The after-school program

served very high risk teenage participants, all of whom: (1) had been suspended or expelled from

school, (2) recorded an average of over 40 days absent or truant during the previous year, (3)

collected at least 20 discipline referrals the previous year and had been assigned to an alternative

school due to aggressive behavior and failure in school, (4) performed at least two grade levels

behind their peers, and (5) came from families meeting federal poverty guidelines. The

intervention consisted of a comprehensive combination of tutoring, group counseling,

enrichment, and social activities. After participating in the program for two years, students’ basic

skill levels improved at least two grade levels, daily attendance increased, discipline referrals

decreased, and none of the students was suspended or expelled. But without a control group, the
 After-School Programs 12

investigators could not disentangle the contributions of the after-school program from those of

the alternative school itself to the observed changes in students’ behavior.

Arbreton, Sheldon, and Herrera (2005) synthesized 20 years of research on the Boys &

Girls Clubs. They found no methodologically rigorous evaluations, such as longitudinal research

and randomized controlled trials. Nevertheless, the authors contended that evaluations showed

Boys & Girls Clubs resulting in: (1) reductions in delinquent behaviors, (2) increased academic

achievement, (3) increased access to and safe utilization of technology, (4) increased career goals,

(5) improved attitudes toward school, (6) more positive relationships, and (7) greater access to safe

places. Similarly, Donaldson and Ronan (2006) found many positive associations between sports

participation and adolescent emotional and behavioral well-being in their analysis of personality

tests administered to 203 young adolescents. The authors cautioned readers about drawing

conclusions regarding the causal role of sports given the correlational nature of their study.

Most research on after-school programs conducted without control groups reported

positive effects. Miller’s (2005) findings were an exception. The Massachusetts After-School

Research Study (Miller, 2005) examined 78 after-school programs in Massachusetts with the aim

of linking program characteristics and program quality. Investigators focused on how the

programs were run – not on outcomes – for assessing program quality. The investigators

administered a pretest and posttest to participants but not to any comparison groups. Participants’

performance (as reported by teachers) varied with students’ background characteristics but not

with program characteristics or the investigators’ measure of program quality.

A recent survey of different ways in which youth spend time after school touted positive

outcomes, yet the article presented data documenting the existence of a potentially confounding

self-selection problem (Reisner et al., 2007). The authors identified 35 high quality after-school
 After-School Programs 13

programs from among 200 candidate programs and used cluster analysis to group students based

on their level of participation. Students fell into four broad categories: (1) participated in a high

quality program plus other after-school activities, (2) participated only in a high quality program,

(3) supervised at home, and (4) not supervised at home and participated irregularly in a high

quality program and other activities. The authors found substantial benefits associated with

participation in the high quality after-school programs. However, the authors reported that

demographic characteristics of students differed among the four clusters. Therefore performance

differences among the clusters may have resulted from the types of students choosing the clusters

and not from the cluster activities.

Some investigators attempted to counter the self-selection problem by using statistical

controls when random assignment was not possible. Jenner and Jenner (2007), for example,

devised a quasi-experimental design for assessing the 21st Century Learning Centers in Louisiana

that included use of standardized test scores to measure program impact. The authors reported a

statistically significant improvement in academic performance for students who participated 30

days or more in the after-school programs. However, participant pretest scores were substantially

lower than those of non-participants, thereby setting up a classic case in which regression to the

mean could be mistakenly interpreted as an intervention effect (Campbell and Kenny, 1999). The

authors neglected to show that the change in participants’ scores was greater than the expected

movement toward the mean score for purely statistical reasons when students with extreme

scores were tested again. Furthermore, the study suffered from two opportunities for self-

selection. First, students selected themselves to participate or not in the after-school programs.

Second, some participants chose to attend the program for thirty days or more, while others

chose to attend less frequently. Those students motivated (themselves or by their parents) to join
 After-School Programs 14

the after-school programs and also attend frequently might have been motivated to improve

academically even without the presence of the programs.

Some reviewers of after-school program evaluations screened out studies they deemed

methodologically weak. For example, Lauer, et al. (2004) required all evaluations they reviewed

to have employed either some form of matching and/or statistical control for comparing program

participants with other students who did not participate. Unfortunately, neither procedure

satisfactorily overcomes the possibility of selection bias having produced the effects attributed to

after-school programs. The investigators examined the impact of programs delivered outside the

school day (not limited to after-school) on academically low-achieving students or students at-

risk for poor academic performance. Overall, out-of-school programs produced small, positive

effects on both reading and mathematics outcomes. The greatest impact occurred with those

reading programs that employed one-on-one tutoring (there were not enough mathematics

studies to examine this factor). Positive effects on reading outcomes were greater for elementary

students than for high school students, while the reverse occurred with mathematics – positive

effects were greater for high school students than for elementary students. Results for other

program characteristics were negative or mixed. Whether the program was delivered after school

or in the summer made no difference with regard to impact. Similarly, whether programs focused

primarily on academic or on academic plus social skills made little difference. The role of

programs’ duration was complex. Programs that lasted more than 45 hours had a greater impact

on both reading and mathematics than shorter programs, but only up to a point. Reading

programs that lasted more than 210 hours and mathematics programs that lasted more than 100

hours resulted in no improvement in outcomes.

 After-School Programs 15

The largest, most ambitious, and among the most rigorous evaluations of after-school

programs conducted so far assessed the impact of the 21st Century Community Learning Centers

program (Dynarski et al. 2004; Dynarski et al. 2003; James-Burdumy et al., 2005). Elementary

and middle school samples were drawn from 3,600 schools located in 903 communities across

the country that had been funded as part of the first three cohorts of the national 21st Century

after-school program grant initiative. One thousand elementary students from 18 over-subscribed

programs were randomly assigned to either an after-school program or a control group in the first

year of the evaluation. Sixteen hundred more elementary students were added for the second

year. At the middle school level, a representative sample of all 21st Century middle school

programs was drawn. The final sample at the middle school level consisted of 4,300 students

from 61 schools. Middle school participants were matched with other students not attending the

after-school programs.

Results from the evaluation of the 21st Century Community Learning Centers must be

weighted heavily because of the rigor of the research and the size of the samples. Positive results

included: (1) elementary participants, but not middle school participants, felt safer during after-

school hours than non-participants; and (2) elementary participants’ parents reported greater

attendance at school events than parents of non-participating students. Negative outcomes

included: (1) after-school programs failed to reduce the number unsupervised students during

after-school hours; (2) few academic effects surfaced, despite the program’s emphasis on

academics; (3) middle school participants exhibited more negative behavior than controls on

some measures and the same amount on others; (4) little evidence of developmental

improvements emerged; (5) over half of the middle school students who could have attended in
 After-School Programs 16

the program’s second year chose not to; (6) attendance was low – roughly 30 days out of the

school year for middle school students and about 60 days for elementary students.

Another randomized trial also found few positive effects of an after-school program.

Unlike most after-school programs, the Cooke Middle School After-School Recreation Program

(CASP) program evaluated by Lauver (2002) took place in the evening from 5 pm to 7 pm,

rather than the afternoon. Lauver randomly assigned 227 applicants to either the program or a

control group and measured outcomes with academic records, achievement test scores, and

surveys. CASP served a disadvantaged middle school located in an urban setting and offered

many activities to students, the most popular of which were dance, martial arts, and basketball.

While a computer lab was available, academics were not stressed. In general, the program was

not highly structured in that students had great flexibility in when to attend and choice of

activities. Parents’ reactions to the program were extremely positive. Parents of participants

believed that the program established a safe place for their children and motivated their children

to spend more time on their homework. Parents also believed that the program resulted in

improved behavior of their children and that it helped their children learn new skills. Students

also responded favorably to the program. They believed their attitudes toward school improved

and they were more likely to have aspirations for post-high school education than control

students. However, the program showed no impact on school attendance, grades, test scores, in-

school behavior, or time spent on homework. As discussed below, these findings are consistent

with Durlak and Weissberg’s (2007) claim that unstructured after-school programs tend to have

little or no observable impact on participants.

 After-School Programs 17

Reports Linking Positive Effects to Program Characteristics

Durlak and Weissberg (2007) linked positive outcomes to the way after-school programs

design their training for youth. On the one hand, the authors’ meta-analysis of 73 after-school

program studies found 39 programs that produced significant, positive personal or social skills

outcomes in seven areas: (1) feelings of self-confidence and self-esteem, (2) school bonding

(positive feelings and attitudes toward school), (3) positive social behaviors, (4) school grades

and achievement test scores, (5) reduction of problem behaviors (e.g., aggression,

noncompliance), (6) conduct problems, and (7) drug use. On the other hand, these positive

outcomes occurred only in programs that employed evidence-based training approaches. Twenty-

seven other after-school programs that did not use evidence-based training approaches reported no

significant improvement in any of the seven outcome categories. The authors’ determination of

which programs employed evidence-based training approaches relied on two criteria related to

the training process and two criteria related to program content. The two training approach

criteria were: (1) the presence of a sequenced set of activities designed to achieve the targeted

skill objectives, and (2) use of active forms of learning. The two program content criteria were:

(1) the presence of at least one program component focused on developing personal or social

skills, and (2) explicit targeting of specific personal or social skills. The authors explored other

possible explanations for their results, such as the role of the scientific rigor with which the study

was conducted (use of randomization, reliability of outcome measures, and problems with

attrition). None of the other examined variables accounted for the major findings – only use of

evidence-based training methods predicted whether or not a program had an impact on youth.

Similarly, the Gottfredson, Cross, and Soulé (2007) article serves a critical function in

demonstrating the direction that research on after-school programs must take in order to make
 After-School Programs 18

progress – that of moving beyond all-or-nothing evaluations to investigating program

characteristics associated with varying levels of effectiveness. Focusing on problem behavior,

Gottfredson, Cross, and Soulé (2007) identified characteristics of 35 Maryland after-school

programs related to victimization, substance use, and delinquent behavior. Reductions in

problem behaviors were associated with structured programming, small program size, more

highly educated staff, and a higher percentage of male staff. The authors emphasized in their

introduction one of the most serious obstacles to conducting research on after-school programs –

that of selection bias. Unfortunately, their study failed to overcome this obstacle. Their

comparison group of students who did not participate in after-school programs differed

significantly from participants on key dimensions of ethnic background, socio-economic status,

and drug use. Furthermore, high attrition from both the participants’ group (41%) and

comparison group (31%) compounded the possibility that selection bias distorted the study

findings.

Do After-School Programs Reduce The Number Of Unsupervised High-Risk Children?

The public’s desire to provide adult supervision during after-school hours for high-risk

children who would otherwise be unsupervised contributed to the growth of after-school

programs. Thus, a basic question when assessing the impact of after-school programs asks

whether these programs significantly increase the number of youth supervised by adults during

the hours following school. Surprisingly, most evaluations of after-school programs neglected

this question. Results from two studies that did address the issue demonstrated that simply

enrolling students in after-school programs does not automatically reduce the number of

unsupervised youth during the hours after the school day ends.
 After-School Programs 19

The large evaluation of the 21st Century after-school program described above yielded the

strongest evidence of after-school programs failing to achieve the goal of reducing the number of

unsupervised youth (Dynarski et al. 2004; Dynarski et al. 2003; James-Burdumy et al., 2005).

This massive study drew its sample of 2,600 elementary and 4,300 middle school students from

across the country and found no reduction in the number of latchkey children resulting from

participation in 21st Century after-school programs.

Another study that addressed the issue provided mixed results. Weisman and Gottfredson

(2001) surveyed parents of children attending their sample of eight Maryland after-school

programs and learned from parent reports that 60% of the children in the programs would have

had adult supervision even if there had been no programs. The programs did provide adult

supervision for a portion (40%) of participants who would otherwise have been unsupervised

after school. The study also found that students who dropped out of after-school programs

exhibited higher levels of risk than students who remained in the programs for a year. Thus, the

primary target population for the after-school programs – high-risk youth – dropped out more

frequently than those students assumed to have less need for the programs.

The Cost of After-School Programs

The rapid expansion in the number of after-school programs has proceeded without

careful examination of the associated costs. In fact, little is known about the costs of after-school

programs, and there may not yet exist even a single effort to contrast the costs with the expected

benefits. Lind, Relave, Deich, Grossman, and Gersick (2006) made a rare foray into the cost

question with their review of efforts to calculate annual per-child costs in after-school programs.

They reported a wide variation in costs – ranging from $449 to $7,160 per student – and

discussed some of the factors accounting for the great disparity. Major factors affecting reported
 After-School Programs 20

after-school program costs included differences among programs in size, how costs were

calculated, whether in-kind resources were taken into account, and whether startup, operating,

and system-building costs were included. The authors observed that information does not yet

exist regarding what benefits accrue from increased expenditures on after-school programming.

For example, no data exist on the impact of greater expenditures for reducing staff-youth ratios

or providing professional development.

DISCUSSION

Most reports of positive after-school program outcomes appear unfounded due to

methodological flaws that negate the positive conclusions. Few evaluators have managed to

incorporate adequate control groups into their research for making comparisons with after-school

participants. Some evaluators claimed positive effects for students attending after-school

programs by reporting pretest-to-posttest changes while ignoring the need for comparison with

other students who did not participate in the programs. Most investigators who employed control

groups either chose inappropriate groups or employed ineffective statistical methods to

compensate for differences between intervention and control groups.

With rare exceptions, several obstacles prevented researchers from employing the types

of research designs that can produce unambiguous results. Typically, after-school programs

invite the entire target population – often all students in certain grades attending a school – to

participate, thereby eliminating the possibility of random assignment to an intervention or

control group. When more students express interest in an after-school program than the program

can accommodate, the opportunity exists for randomly assigning applicants to either participate

or not in the program. This was the situation that occurred in the evaluation of the 21st Century

Learning Centers Program (Dynarski et al. 2004; Dynarski et al. 2003; James-Burdumy et al.,
 After-School Programs 21

2005). Other possibilities for random assignment exist but require vast financial resources to

carry out the study and difficult-to-obtain cooperation from many schools. A few researchers in

other prevention areas, such as substance abuse and smoking, managed to randomly assign

schools, or even school systems, to either intervention or control conditions (Botvin, Baker,

Dusenbury, Botvin, & Diaz, 1995; Peterson, Kealey, Mann, Marek, & Sarason, 2000). Such an

endeavor has not yet been undertaken with respect to after-school programs.

Infrequent attendance and high attrition levels create selection bias even if investigators

managed to establish a randomized control experiment. Some studies that began with a full

complement of participants ended up reporting on the performance of only a select few who both

remained in the after-school program and also attended frequently. These remaining students did

not represent the initial samples of students who began attending the after-school programs.

Results from studies that did employ rigorous research designs failed to detect evidence

of program impact. Two randomized trials showed little or no advantage incurred by participants

in after-school programs. Dynarski and colleagues randomly assigned 2,600 elementary students

to participate or not in 21st Century Learning Center after-school programs across the country

(Dynarski et al. 2004; Dynarski et al. 2003; James-Burdumy et al., 2005). Participation

accomplished neither of the two most important goals – the number of students without adult

supervision during the hours after school remained unchanged, and academic performance of

participants showed no improvement when compared with controls. Similar outcomes occurred

for the middle school sample in the study. However, those results carry much less weight due to

the absence of a randomized trial research design.

Another strong evaluation yielded similarly negative results. Lauver (2002) randomly

assigned middle school students to either a program offering many activity options or to no
 After-School Programs 22

program participation. Parents strongly supported the program, but measures of school

attendance, grades, test scores, in-school behavior, and time spent on homework showed no

advantage for participants over controls

The most promising analysis of after-school programs suggested that after-school

programs can be effective – but only if they meet certain conditions. After-school programs

demonstrated positive outcomes in Durlak and Weissberg’s (2007) analysis of 73 evaluations

only if they incorporated evidence-based training methods. More specifically, effective programs

employed structured educational procedures involving active forms of learning and targeted

specific personal or social skills.

Given the paucity of rigorous evaluations, conclusions drawn regarding the impact of

after-school programs must be drawn cautiously and tentatively. First, Durlak and Weissberg’s

(2007) research showed that after-school programs improved personal and social skills when

using appropriate training methods that focused on those specific skills. Can after-school

programs improve other skills? The 21st Century Learning Centers that Dynarski and colleagues

evaluated failed to improve academic skills, even though those were the skills on which they

focused (Dynarski et al. 2004; Dynarski et al. 2003; James-Burdumy et al., 2005). But

extenuating circumstances question such a conclusion. The programs had little opportunity to

influence students, since most participated infrequently and spent only one year in the program.

Two conclusions about the effects of after-school programs seem warranted at this point.

A combination of (1) specific goals, (2) structured programming based on sound educational

techniques, and (3) frequent attendance can produce impressive gains among participants.

Conversely, after-school programs are unlikely to produce desired results due to any of the
 After-School Programs 23

following: (1) absence of clear goals, (2) absence of structured programming, or (3) infrequent

attendance.

An objective assessment of the role that after-school programs play must also examine

the extent to which these programs reduce the number of at-risk youth without adult supervision

during the hours following school. Funding for after-school programs increased in part due to

concerns about the large number of latchkey children and evidence that juvenile crime peaked

between 2 p.m. and 6 p.m. on school days. Yet Dynarski and colleagues (Dynarski et al. 2004;

Dynarski et al. 2003; James-Burdumy et al., 2005) found no reduction in the number of

unsupervised children following school resulting from the introduction of the 21st Century

Learning Centers after-school programs. Weisman and Gottfredson (2001) reported that 40% of

the participants gained adult supervision they would not otherwise have had. But another finding

of theirs raised the question of how long supervision provided by the after-school programs

lasted. Students judged to be at-risk dropped out of the programs more frequently than others. In

sum, one of the most important unknowns regarding after-school programs remains the question

of the extent to which they reduce the number of unsupervised high-risk students following the

end of the school day

 After-School Programs 24

REFERENCES

Arbreton, A. J. A., Sheldon, J, & Herrera, C. (2005). Beyond safe havens: A synthesis of 20 years

of research on the Boys & Girls Clubs. Philadelphia, PA: Public/Private Ventures.

Biglan, A., Metzler, C. W., Wirt, R., Ary, D., Noell, J., Ochs, L., French, C. & Hood, D. (1990).

Social and behavioral factors associated with high-risk sexual behaviors among

adolescents. Journal of Behavioral Medicine, 13, 245–261.

Block, J., Block, J. H., & Keyes, S. (1988). Longitudinally foretelling drug usage in adolescence:

Early childhood personality and environmental precursors. Child Development, 59,

336–355.

Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M. & Diaz, D. (1995). Long-term follow-up

results of a randomized drug abuse prevention trial in a white middle-class population.

Journal of the American Medical Association, 273, 1106-1112.

Bureau of Labor Statistics. (2000). Families with own children: Employment status of

parents by age of youngest child and family type, 2001-2002 annual averages.

Washington, DC: U.S. Department of Labor.

Campbell, D. T. & Kenny, D. A., (1999). A primer on regression artifacts. New York: The

Guilford Press.

Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for

research on teaching. In N. L. Gage (Ed,) Handbook of research on teaching (pp 171-

246). Chicago: Rand McNally. [Reprinted in 1966 as Experimental and quasi-

experimental designs for research. Boston: Houghton Mifflin.]

Cernkovich, S.A. & Giordano, P.C. (1987). Family relationships and delinquency. Criminology,

25, 295–319.
 After-School Programs 25

Chung, A. (2000). Working for children and families: Safe and smart after-school programs

Department of Justice, Washington D.C; Department of Education, Washington,

D.C.: Government Printing Office.

Dishion, T. J., Patterson, G. R., Stoolmiller, M. & Skinner, M.L. (1991). Family, school, and

behavioral antecedents to early adolescent involvement with antisocial peers.

Developmental Psychology, 27, 172–180.

Donaldson, S. J. & Ronan, K. R. (2006). The effects of sports participation on young

adolescents’ emotional well-being. Adolescence, 41, 369-389.

Durlak, J. A. & Weissberg, R. P. (2007). The impact of after-school programs that seek to

promote personal and social skills. Chicago, IL: The Collaborative for Academic, Social

and Emotional Learning.

Dwyer, K. M., Richardson, J.S., Danley, K.L., Hansen, W.B., Sussman, S.Y., Brannon, B., Dent,

C.W., Johnson, C.A., & Flay, B.R. (1990). Characteristics of eighth-grade students who

initiate self-care in elementary and junior high school. Pediatrics, 86, 448-454.

Dynarski, M., Moore, M., James-Burdumy, S., Rosenberg, L., Deke, J. & Mansfield, W. (2004).

When schools stay open late: The national evaluation of the 21st century community

learning centers program: New findings. Report submitted to the U.S. Department of

Education. Princeton, NJ: Mathematica Policy Research, Inc.

Dynarski, M., Moore, M., Mullens, J. Gleason, P., James-Burdumy, S., Rosenberg, L., Pistorino,

C., Silva, T., Deke, J., Mansfield, W., Heaviside, S., & Levy, D. (2003). When schools

stay open late: The National Evaluation of the 21st Century Community Learning

Centers Program: First year findings. Report submitted to the U.S. Department of

Education. Princeton, NJ: Mathematica Policy Research, Inc.

 After-School Programs 26

Fagan, A. A. (2007). Investment in effective after-school programs: Is it worth the risk?

Criminology & Public Policy, 6, 319-326.

Fashola, O. S. (1998). Review of extended-day and after-school programs and the effectiveness

(Report No. 24). Baltimore, MD: Center for Research on the Education of Students

Placed at Risk.

Gottfredson, D. C., Cross, A., & Soulé, D. A. (2007). Distinguishing characteristics of effective

and ineffective after-school programs to prevent delinquency and victimization.

Criminology & Public Policy, 6(2), 289-318.

Gottfredson, D. C., Gottfredson, G. D., & Weisman, S.A. (2001). The timing of delinquent

behavior and its implications for after-school programs. Criminology & Public Policy,

1, 61–86.

Grossman, J. B., Price, M. L., Fellerath, V., Jucovy, L. Z., Kotloff, L.J., Raley, R., & Walker, K.

E. (2002). Multiple choices after school: Findings from the extended-service schools

initiative. Philadelphia, PA: Public Private Ventures.

Hofferth, S.L., Brayfield, A., Deich, S., & Holocomb, P. (1991). National Child Care Survey,

1990. Washington, DC: Urban Institute Press.

James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W. & Pistorino, C. (2005).

When schools stay open late: the national evaluation of the 21st century community

learning centers program: Final report. Report submitted to the U.S. Department of

Education. Princeton, NJ: Mathematica Policy Research, Inc.

Jenner, E. & Jenner, L. W. (2007). Results from a first-year evaluation of academic impacts of an

after-school program for at-risk students. Journal of Education for Students Placed at

Risk, 12, 213-237.

 After-School Programs 27

Kane, T. J. (2004). The impact of after-school programs: Interpreting the results of four recent

evaluations. New York: W. T. Grant Foundation.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M.

(2004). The effectiveness of out-of-school-time strategies in assisting low-achieving

students in reading and mathematics: A research synthesis. Washington, DC: Department

of Education, Institute of Education Sciences.

Lauver, S. C. (2002). Assessing the benefits of an after-school program for urban youth: An

impact and process evaluation. Unpublished doctoral dissertation, University of

Pennsylvania, Philadelphia.

Lind, C., Relave, N., Deich, S., Grossman, J. & Gersick, A. (2006). The costs of out-of-school-

time programs: A review of the available evidence. New York, NY: The Wallace

Foundation.

Long, S. & Clark, S. (1997). The new child care block grant: State funding choices and their

implications. Washington, DC: Urban Institute.

Martin, D., Martin, M., Gibson, S. S., & Wilkins, J. (2007). Increasing prosocial behavior and

academic achievement among adolescent African American males. Adolescence,

42, 689-698.

National Institute on Out-of-School Time. (2003). Fact sheet on school-age children’s out of

school time. Wellesley, MA: Wellesley College, Center for Research on Women.

McComb, E.M. & Scott-Little, C. (2003). After-school programs: Evaluations and Outcomes.

Greensboro, NC: The Regional Educational Laboratory at SERVE.

Miller, B.M. (2005). Pathways to Success for Youth: What Counts in Afterschool.

Boston, MA: United Way of Massachusetts Bay.

 After-School Programs 28

Newman, S. A., Fox, J. A., Flynn, E. A., & Christeson, W. (2000). America’s after-

school choice: The prime time for juvenile crime or youth enrichment and

achievement. Washington, DC: Fight Crime: Invest in Kids.

Osofsky, J.D. (1999). The impact of violence on children. The Future of Children, 9(3),

33-49.

Peterson, A. V., Kealey, K. A., Mann, S. L., Marek, P. M. & Sarason, I. G. (2000). Hutchinson

smoking prevention project: Long-term randomized trial in school-based tobacco use

prevention—results on smoking. Journal of the National Cancer Institute, 92, 1979-1991.

Posner, J. K. & Vandell, D. L. (1999). After-school activities and the development of

low- income urban children: A longitudinal study. Developmental Psychology, 35,

868-879.

Reisner, E. R., Vandell, D. L., Pechman, E. M., Pierce, K. M., Brown, B. B, & Bolt, D. Charting

the benefits of high-quality after-school program experiences. Washington, DC: Policy

Studies Associates, Inc.

Richardson, J. L., Dwyer, K., McGuigan, K., Hansen, W. B., Dent, C., Johnson, C. A.,

Sussman, S. Y., Brannon, B., & Flay, B. (1989). Substance use among eighth-

grade students who take care of themselves after school. Pediatrics, 84, 556-566.

Riggs, N. R. & Greenberg, M. T. (2004). After-school youth development programs: A

developmental-ecological model of current research. Clinical Child and Family

Psychology Review, 7, 177-190.

Scott-Little, C., Hamann, M. S. & Jurs, S. G. (2002). Evaluations of after-school

programs: A meta-evaluation of methodologies and narrative synthesis of

findings. American Journal of Evaluation, 23, 387-419.

 After-School Programs 29

Snyder, H., Sickmund, M., & Poe-Yamagata, E. (1996). Juvenile offenders and victims:

1996 update on violence. Washington, DC: Office of Juvenile Justice and

Delinquency Prevention.

Walker, K. E. & Arbreton, A. (2004). After-school pursuits: An examination of outcomes in the

San Francisco Beacon Initiative. Philadelphia, PA: Public/Private Ventures.

Weisman, S. A. & Gottfredson, D. C. (2001). Attrition from after school Programs:

Characteristics of students who drop out. Prevention Science, 2, 201–205.

Zief, S. G., Lauver, S., & Maynard, R. A. (2004). Impacts of after-school programs on

student outcomes: A systematic review for the Campbell collaboration. C2

Reviews of Interventions, and Policy Evaluations. May 9, 2008.

http://www.campbellcollaboration.org/doc-pdf/zief_afterschool_review.pdf

View publication stats