Executive Summary ae Children in Philadelphia are at risk. Over one-fifth of children in Philadelphia live in poverty, and the lived experience of povezty is about more than limited access to material resources. It includes dangerous neighborhoods, strained families and stressed communities. Thankfully, many children ‘who are exposed to tisk go on to success in school and life, demonstrating the “ordinary magic” of resilience (Masten, 2001, p. 227). But in today’s Philadelphia, resilience must come from the child’s internal resources and those of their family. ‘The most powerful engine of resilience, opportunity, and equality ~ public education is in serious jeopardy. The result is the sad irony that the children who would benefit most from access to high-quality educational opportunities are also those least likely to receive them. Into this breach stepped Play on Philly, guided by the essential theory of change this evaluation, assesses: that access to high-quality music education can mitigate the effects of risk on academic achievement and prosocial behavior by fostering pasticipants’ executive functions. Neacly every program that works with children at risk claims ancillary benefits; few can substantiate these claims, ‘and fewer still can articulate Aum theie program achieves these effects. In this respect, Play on Philly is unique, and it is part of what makes Play on Philly a leader in the field of Bi Sistema-inspired programs, ‘This theory of change is summarized in the following model, which yielded the four questions that guided the evaluation. Figuse SI Academic Achievement Eogagement ip Executive | Paay-on Phity Punecions Guiding Question 1. Do children ia Play on Philly exhibit higher levels of academic achievement? “The answer appears to be yes. Students enrolled in Play on Philly exhibited: © Higher standardized test scozes, both in general and for each of three specific disciplines (eading, language, and math; see figure S2) = Hligher year-end grades in English language arts. Larger, positive changes in math scores from the fist co the third trimester. An Exaluation of Play on Phil 3In cach case these differences were substantial. Enrollment in Play on Philly was associated with composite standardized test scores that were, on average, ten points higher than scores for students who were not enrolled. In other words, a student who achieved a standardized test score in the 70* percentile without Play on Philly could be expected to score above the 80 percentile with Play on Philly Figure $2 «20 610 -— — — oo - 390 * 580 510 Comparison 360 > = pop Standardized Test Score 550 540 -— 530 520 Composite Reading Language Math Students in 4 through 8% grade who were enrolled in Play on Philly exhibited year-end English language arts grades that were half a letter grade higher than their peers; these students also exhibited changes in math grades that were approximately half a letter grade larger than the changes exhibited by their peers. ‘Thus, if students who were not ensolled in Play on Philly improved from a C+ t0 a B- over the course of the school year, stadeats in Play on Philly improved from a C+ to a B. Similar, ‘but smaller effects were observed when Play on Philly was considered as a continuous phenomenon: change in seores on the assessment of Fundamental musical knowledge was associated with positive change in math grades from the first to the tied tsimester that were approximately one-quarter of letter grade larger than the change for less engaged peers. ‘Thece things must be borne in mind when considering these results: ‘+ First, students were entolled in Play on Philly at random. That is, students first elected to be in Play on Philly, and were then assigned to be in the program by lottery (sce Methods section below). ‘Therefore we may conclude that any differences we observe between students enrolled in Play on Philly and their peers are attributable to the progeam, rather than some pre-existing difference between the groups. An Eraluation of Play on Phily 4+ Second, all results control for students’ gender, ethaicity, and classroom. Thus the Uifferences in standardized test scores, grades, or any other outcomes that could be atusiburable to these faetors are removed ftom the equation before considering the association of Play on Philly wth these outcomes + Third, because some of ous outcomes are correlated with one another (scores on the individual domains of standardized tests, for example), we employed appropriate corrections to guard against the chance we would observe results when in fact none were present. |All this is to say that the results presented ia this executive summary and in the body of the report itself are only those that met the stringent burden of proof of that would be employed in the process of peer-seview. Guiding Question 2. Do children in Play on Philly exhibit higher levels of prosocial behavior? "There was also evidence that engagement in Play on Philly was associated with higher levels of prosocial behavior, in that enrollment in Play on Philly was associated with fewer absences from school (see figure $3). ‘The size of the difference corresponded to approximately two fewer absences for students enrolled in Play on Philly over the course of the school year. While this may not appear to be a large difetence, in 2012-13 students at St. Francis missed, on average, fewer than 7 days of school. ‘Thus the effect associated with enrollment in Play on Philly represents a reduetion in’ absenteeism of approximately 30%, Among students in Play on Philly, engagement as rated by Play on Philly instructors was related to higher year-end ratings of personal and social skills by students’ school teachers. Every point increase on the teacher-engagement scale corresponded to an increase on the personal and social skills ratings of approximately one-tenth of a point, ‘Therefore the difference in personal and social skills grades between a very unengaged student in Play on Philly, with a teacher-rated engagement of 2 (out of 14), and highly-engaged student, with an engagement of score of 12, would be approximately 1 point (out of 4), or a 25% difference within the metric of the scale, Figure $3 6 4 2 Comparison 254 ais 5 pop a An Evaluation of Play o» Pilly 5Guiding Question 3, Do children in Play on Philly exhibit better performance on tasks of executive functions? Executive functions are a set of cognitive processes that allow an individual to organize their behavior towards the achievement of goals. Itis therefore not surprising that executive functions have been linked to academic achievement and prosocial behavior. Engagement in Play on Philly ‘was related to better performance on four of the nine tasks of executive functions. Two of these tasks wete designed to assess selective attention. While the other two tasks putatively assessed, working memory and behavioral inhibition, the case can be made that each of these tasks placed considerable demands on attention as well (see Discussion). Across tasks, it appeared that students enrolled in Play on Philly were substantially more efficient ia directing their attention as needed during the course of each task. ‘The final set of analyses revealed that this increased efficiency may, in part, underlie some of the differences in academic achievement that were observed, Guiding Question 4, To what extent do executive functions account for the effects of Play on Philly on academic achievement and/or prosocial behavior? ur analyses revealed compelling evidence that one of the ways in which Play on Philly confers benefits to academic achievement is through improvements in executive functions, and, in particular, by fostering the capacity for selective atention. The sclationship between engagement in Play on. Philly and composite standardized test scores was mediated by three of the four measures of executive functions for which better performance was associated with enzollment. When each ‘measure of executive function is considered, the strength of the direct relationship between ‘engagement in Play on Philly aad standardized test scores is reduced between 5 and 40%, indicating that executive Functions account for 5 to 40% of the effects of Play on Philly on academic achievement (as diageammed in figure $4) Figure S4 ww Memory Span An Balaton of Play on Philly 6In the Discussion section below we outline a number of ways in which we hope to build upon these results in the cusrent phase of the evaluation. Perhaps most important among these is our desire 0 expand the set of possible mediating mechanisms beyond executive functions, while improving our ability to measure change in behavioral outcomes which, while genuine, may not be manifest in as blunt a measuse as school absences. Figure S5 presents the expanded logic guiding the current phase of the evaluation, Figuse SS Perseverance Achievement Play on Philly is a program that inspires hope. Being at one of the sites, and watching children who, absent Play on Philly’s existence, might never see a cello, learn to play one is a life-affirming, experience. Observing this process, one begins to suspect that the experience is also making classical music relevant to children and families who might otherwise think little of it. Tn an ideal world, the ‘musical education of children and the perpetuation of an art would be enough to ask of any organization, In the real world, funders demand that ancillary benefits be realized. Fortunately, we can say with confidence that engagement in Play on Philly is associated with genuine and positive differences in academic achievement, behavior, and some of the more fundamental cognitive processes that underlie performance across a host of domains. Invrsperonal ‘sille Program Musical espenene Accomplishment Resiience Executive iH An Bualuation of Play on Philly