Effect of Project-Based Learning On High School Students' State-Mandated, Standardized Math and Science Exam Performance

Received: 10 January 2018 Revised: 8 May 2019 Accepted: 1 June 2019
DOI: 10.1002/tea.21582
RESEARCH ARTICLE |
Effect of project-based learning on high school

students' state-mandated, standardized math and
science exam performance
Tara T. Craig1 | Jill Marshall2
1
Coastal Carolina University, Department of Mathematics and Statistics, Conway, South Carolina
2
The University of Texas at Austin, Department of Curriculum and Instruction, Austin, Texas
Correspondence Abstract
Tara T. Craig, Coastal Carolina University, In a technologically driven society, math and science stu-
Department of Mathematics and Statistics,
dents in the United States are falling further and further
Conway, SC.
Email: taracraig01@gmail.com behind their international counterparts, resulting in an
influx of STEM focused, reformed K-12 schools, includ-
ing schools focused on project-based learning (PBL). This
article reports a study of the effectiveness of PBL on high
school students' performance on state mandated standard-
ized mathematics and science achievement measures.
Manor New Tech High School is a nationally recognized
model STEM school, with a diverse student population,
where all instruction is delivered through PBL. Although
there is ample research suggesting that PBL is advanta-
geous for increasing STEM learning compared to conven-
tional teaching approaches, there is a lack of studies
randomly assigning students to receive PBL. Further, some
of the effects observed for students attending project-based
schools could be due to a self-selection bias for students or
parents that choose such an alternative learning environ-
ment. This study addresses both of these concerns and
found that students taught through PBL, as a group, mat-
ched performance of conventionally taught students on
all science 11th grade and mathematics 9th, 10th, and
11th grade TAKS achievement measures and exceeded
performance by a scale score increase of 133 for the 10th
J Res Sci Teach. 2019;56:1461–1488. wileyonlinelibrary.com/journal/tea © 2019 Wiley Periodicals, Inc. 1461
1462
| CRAIG AND MARSHALL
grade science TAKS measure by (B = 133.082, t = 3.102,

p < .05). One possible explanation of the differences
observed in this study could be the TAKS instrument used
to capture student math and science achievement that inter-
prets “real-life applications” of content differently between
math and science questions. These results align with litera-
ture on the effects of PBL and deepen our understanding
of these effects by providing a controlled study with ran-
dom assignments to the PBL experience. Future research
looking at the effect of PBL on achievement on the PISA
could be beneficial in identifying benefits of PBL imple-
mentation in schools.
KEYWORDS
achievement, mathematics education, project-based learning, quasi-
experimental, school choice, science education, selection bias,
standardized tests, STEM education
1 | INTRODUCTION
In this study, we compared high school math and science performance of students who applied to a
project-based learning (PBL) high school and were admitted in a random lottery with that of a mat-
ched, control group of students that applied to the school but were waitlisted, and in turn attended
the conventional, lecture-focused high school in the same district. Our goal was to expand the body
of empirical research on the effects of PBL on high school performance outcomes. By taking advan-
tage of the opportunity the school lottery provided for random assignment to the PBL treatment
group, we were able to provide evidence of the effectiveness of PBL in mathematics and science at
the high school level, at least for students who would, along with their parents, elect such an alterna-
tive schooling environment. In the United States, this is an increasingly larger percentage of the
population.
In the 21st century, students in the U.S. are reportedly lagging behind international students in
STEM achievement. In 2012, among the 34 countries in the Organization for Economic Co-operation
and Development (OECD), the United States ranked 27th in mathematics performance on the Pro-
gramme for International Student Assessment (PISA) and 20th in science (OECD, 2012). The
implied need for reform in STEM education has resulted in an influx of alternative schooling options
for K-12 students, from charter schools, to STEM-focused schools, to schools using innovative
teaching methods (PBL in particular), all with the mission of increasing students' achievement in sci-
ence and mathematics (New Tech High Foundation, 2010, Buck Institute for Education, 2019). There
is a broad base of research support for some of the reform options. However, some speculate that the
self-selection that occurs when students and parents are provided school choice results in a popula-
tion of students that are already academically driven or have a supportive family (Cullen, Jacob, &
Levitt, 2005; Murnane, Newstead, & Olsen, 1985). It is possible that the initiative demonstrated in
applying to an innovative school, such as the site used for this research, could impact students'
CRAIG AND MARSHALL
| 1463
success in high school regardless of the school attended, and students that apply to the school would
be as successful in any high school environment. Therefore, it is important to control for this self-
selection bias when looking at the effect of PBL on student learning where students were not ran-
domly provided the opportunity for this type of instruction.
2 | CONVENTIONAL LEARNING
The comparison group explored in this study received conventional instructional methods in their
math and science classrooms at the main high school in the district (Manor ISD, 2012), emphasizing
drill to mastery and test preparation. The authors define conventional instruction as focusing primar-
ily on teacher-centered practices, such as lecture and direct forms of instruction that are focused on
replication and computation instead of inquiry and discovery (Prince & Felder, 2006). Specific to
mathematics, conventional instruction is viewed as procedure-oriented math instruction, focused pri-
marily on replicating teacher-modeled computations (Boaler, 1998). Specific to science instruction,
conventional instruction often reveals itself in the use of laboratory books that provide students with
a step-by-step “recipe” to follow, absent of authentic inquiry and exploration (Hofstein &
Lunetta, 2004).
3 | PROJE CT -B ASE D L E A R N I N G
Project-based learning traces its origin back to Rome and Paris in the late 16th century as architecture
evolved from a vocation to a scholastic subject. Architects of the time wanted professional training
that met the demands of a growing field at the epicenter of science, art, and design. Training based
solely in lecture lacked the contextual components required in the field, resulting in an addition of
the curriculum that included challenges to design churches, monuments, or palaces, thus providing a
context for students to apply the content taught during lectures or workshops (Knoll, 1997). Then in
the late 18th century, engineering education found the same need for instructional methods that com-
bined theory and practice, bringing learning through projects from Europe to America into many
technical universities. The project method of instruction continued to spread to industrial and manual
training schools and general public schools by the late 19th century. The home project movement
began in the early 20th century, focusing on the “authentic task” as the vehicle for learning, building
off Dewey's theory of learning through experience (Knoll, 1997). The movement seized the imagina-
tion of American educators, resulting in 60,000 requests for reprints of the Kilpatrick (1918) article
in Teachers College Record (Kliebard, 2004). Although developed in his name, Dewey felt that pro-
jects, as defined by Kilpatrick, lacked the “enterprise of the child” essential to experiential learning
theory (Dewey, 1938; Knoll, 1997). The movement was quickly seen as too random in its educa-
tional effects to ensure that students learned what they needed to in order to satisfy the education
establishment, college admission criteria in particular, and lost prominence.
The most recent resurgence in project-based education has sought to address this issue by focus-
ing on the need to meet specific educational standards. Learning theorists invoke careful attention to
standards as an essential characteristic of the current instantiation of PBL. For example, Barron et al.
(1998) list “learning appropriate goals” first in their four design principles to ensure “doing with
understanding” as opposed to “doing for the sake of doing,” as previous PBL had been deemed
(p.273). Krajcik and Blumenfeld (2006) also include the fact that students “learn and apply important
1464
ideas in the discipline” while engaged in situated inquiry toward answering a driving question among
the five key features of PBL environments (p.318).
The term “project” in modern education is nothing new or innovative, as veteran teachers will tell
you that they have used projects as a summative assessment and authentic application of content
learned in courses for years. Thomas (2000) explains that what is unique in this new instance of PBL
is that “projects are the curriculum” (p. 3). The site for this research used The Buck Institute for Edu-
cation's definition that an authentic project must include the following eight essential elements:
(1) Begins with a driving question or challenge that provides context and drives instruction,
(2) Aligns with significant content learning goals, (3) Incorporates 21st century skills, (4) Facilitates
in-depth inquiry that allows the student to explore the content, (5) Multiple opportunities providing
student choice and voice, (6) Provides multiple opportunities for self-critique and assessment, and
(8) Results in presenting a final product to a community audience (Larmer & Ross, 2009; BIE,
2019). Therefore, the authors chose to also define authentic project-based learning this way and only
included articles in the literature review where the definition of PBL closely matched these criteria.
With the increased emphasis on significant content, one would hope to see a signature of success-
ful PBL in measures directly related to established standards—standardized test scores in particular.
Further, with its emphasis on learning in contexts relevant to students and their daily lives, one would
expect PBL to enable students who have not traditionally been successful in STEM to engage in such
a way as to enhance their success in these subjects. Thus, PBL also presents the hope of increased
equity in STEM education. There is indeed some evidence for both these effects (Boaler, 1998; Geier
et al., 2008; Moll, Amanti, Neff, & Gonzalez, 1992); however, randomized, controlled studies of the
effect of PBL, particularly at the high school level in America, and especially in mathematics as
opposed to science, are still wanted.
Project-based curriculum builds on students' interest by definition, engaging them in authentic,
relevant challenges (Blumenfeld et al., 1991). Not unexpectedly, therefore, students that participate
in PBL show enhanced motivation to learn content (Mills & Treagust, 2003). PBL provides students
with autonomy over their learning by providing them with choices regarding the process for solving
the driving question or challenge. Choice promotes intrinsic motivation, resulting in greater persis-
tence, and allowing students to pursue challenges in ways that are appropriate for them, while at the
same time inspiring them to take on greater challenges. By permitting, or in some cases requiring,
students to make choices, PBL leverages interest to promote deep, conceptual pursuit of learning
(Bell, 2010).
Despite the paucity of experimental evidence, PBL has continued to spread, with the New Tech
Network (http://www.newtechnetwork.org) as a primary example of the project-based movement. At
New Tech schools, the core curriculum is delivered through PBL with all projects following a back-
ward design, aligning all project units with the required state content standards, and meeting the defi-
nition of PBL given above (New Tech High Foundation, 2010).
The study reported here investigated the effect of attending New Tech Foundation's flagship,
project-based high school, Manor New Technology High School (MNTH), on student math and sci-
ence achievement, as measured by Texas state-mandated standardized tests. There are no academic
admission requirements for students interested in attending MNTH. Any interested eighth grade stu-
dent, or their parent, can provide the student name, gender, and contact information to be placed in
the random lottery for admission. Most students that were not accepted to attend the project-based
MNTH, instead attended Manor High School (MHS) that, at the time of the study, incorporated con-
ventional math and science curriculum (see Figure 1).
CRAIG AND MARSHALL
| 1465
F I G U R E 1 A diagram of the quasi-experimental research design showing the self-selected sample, random
assignment, PBL treatment and conventional control group, and analysis [Color figure can be viewed at
wileyonlinelibrary.com]
The existence of a project-based school with a pool of waitlisted students (i.e., students who had
applied to the school but were not randomly selected in a lottery for admission) that instead attend a
conventional instruction high school in the same district, provided a unique opportunity for a “natu-
ral” experiment. Although the situation did not permit a fully generalizable study since students or
their parents opted to apply to the school, it did allow for a well-matched control group, so that the
effects of PBL, at least on students that would opt for this approach, could be measured. Further, the
design mitigates a possible self-selection effect on the part of academically driven students with sup-
portive families, which has been posited to explain higher achievement in some selective school
environments (Cullen et al., 2005; Murnane et al., 1985). We sought to address the following
research question:
RQ- How does project-based curriculum affect students' mastery of state-mandated math
and science content standards as assessed by standardized tests?
Although admission to the project-based MNTH was random, once students put their name in the
lottery, an a priori analysis was performed to determine whether student characteristics or prior
achievement varied for the control and the PBL samples, demonstrating the “equivalence of the
analytic intervention and comparison groups” as required to meet criteria for strong evidence
(What Works Clearinghouse [WWC, 2002]). Multiple linear regression models were created using
data from both groups, controlling for student characteristics shown to have significant difference
between the control and experimental group determined through statistical testing conducted
a priori.
1466
4 | CONCEPTUAL FRAMEWORK
The theoretical frame for this work (see Figure 2) rests on (a) the foundation of situated cognition,
the recognition that learning is context-dependent and enhanced by an authentic task, and (b) the
expectation that inductive teaching, specifically project-based learning, is a vehicle for enhanced
STEM learning in the secondary curriculum (Boaler, 1998; Bransford, Sherwood, Hasselbring, &
Williams, 1990; Brown, Collins, & Duguid, 1989). Just as the early proponents of project-based
instruction felt that it was crucial to look to the immediate needs and interests of students in order to
engage them in the “hearty purposeful act” (Kilpatrick, 1918, p. 320), in the modern instantiation
projects are embedded in a context and focus on a purpose designed to be of interest to students.
Thus PBL, as the intersection of contextualized, interest-driven instruction and inductive teaching
methods, is posited to enhance achievement, particularly for students for whom intangible, isolated,
learning of mathematics, and science has not been successful.
5 | INDUCTIVE TEACHING METHODS
Conventional teaching and learning follows a teacher-centered, lecture design in which the teacher
provides content and knowledge to students. The main motivation for the students to learn the con-
tent is simply that the teacher explains that they need to learn new content, with a promise that this
content will be applicable to their lives after school (Prince & Felder, 2006). Students are more moti-
vated to learn when the content is situated in solving a problem or challenge because the student will
develop a need to learn the content in order to successfully solve the problem. “As the students
attempt to analyze the data or scenario and solve the problem, they generate a need for facts, rules,
procedures, and guiding principles, at which point they are either presented with the needed informa-
tion or helped to discover it for themselves” (Prince & Felder, 2006, p. 123).
Multiple inductive teaching and learning methods exist, in addition to PBL, in hopes of providing this
need of learning new knowledge to the student. While motivating the learner through providing a need
to learn, inductive teaching and learning methods are always (a) learner-centered focusing the responsi-
bility to learn on the student instead of the teacher, (b) based on research that suggests students learn by
making connections to their existing cognitive structures, and (c) come from a constructivist view point
that students construct new knowledge instead of simply absorbing it (Prince & Felder, 2006). Among
MATH & SCIENCE

STANDARDIZED TEST ACHIEVEMENT
STEM Learning
F I G U R E 2 A schematic of the conceptual

Inductive Relevant framework showing how PBL, at the intersection of
Teaching Context inductive teaching methods and situated in relevant
PBL
context, promotes STEM learning, and in turn STEM
SITUATED COGNITION achievement [Color figure can be viewed at
wileyonlinelibrary.com]
CRAIG AND MARSHALL
| 1467
these inductive instructional methods are inquiry-based learning, PBL, problem-based learning
(PrBL), and discovery learning (Prince & Felder, 2006).
PrBL is an inductive teaching method in which students work in groups to develop a plan and
solve open-ended, authentic problems. Although similar to project-based learning, one main differ-
ence between the two instructional approaches is that PrBL does not meet the essential criteria for
PBL: it is usually a smaller scope (1–3 days), and is not explicitly aligned with learning outcomes
(Prince & Felder, 2006). Literature on the effects of PBL on student learning was reviewed thor-
oughly, but it was necessary to also review literature for similar teaching methods (e.g., PrBL) in
order to situate the PBL results in the larger inductive paradigm.
6 | RESEARCH ON PBL AND STEM LEARNING
There is a broad base of research on the effect of PBL on STEM learning. Some studies have found
positive effects, particularly on measures capturing deep, meaning-oriented learning as opposed to
surface, memorization-intensive, learning (Prince & Felder, 2006). For example, students engaged in
a chair design project that aligned with learning specific content goals around topics in geometry
(e.g., measurement, scale, and perspective) showed significant gains in their understanding of geome-
try concepts as measured by a traditional, multiple choice assessment (Barron et al., 1998). These
gains were statistically significant while controlling for prior mathematics achievement. Rivet and
Krajcik (2004) engaged students in a 4-year project-based science curriculum and observed a signifi-
cant effect between scores on pre- and post-assessments focusing on both deep and surface level
learning. Mergendoller, Maxwell, and Bellisimo (2006) explored the effectiveness of PBL on student
achievement in microeconomics and found that engagement in PBL did not harm student learning
overall and in some cases enhanced performance on assessments.
The majority of the literature on PBL compared to conventional instruction has been focused on
achievement as measured by short-term, research-designed assessments. There are exceptions to this;
in these cases PBL has generally been shown to increase or at least maintain students' learning of
math and science content as assessed on state-mandated and national-level standardized assessments
when compared to lecture-based instruction (Boaler, 1998; Geier et al., 2008; Holmes & Hwang,
2016; Schneider, Krajcik, Marx, & Soloway, 2002). Boaler (1998) conducted a 3-year, longitudinal
study of ~300 students in two secondary schools in England where one school taught math tradition-
ally, while the other taught through PBL. After 3 years of either PBL or conventional curriculum,
students attending the project-based secondary school scored significantly higher on England's
national standardized math assessment.
Han, Capraro, and Capraro (2015) found that low performing students taught through PBL for
3 years had significantly higher growth in achievement on standardized mathematics exams com-
pared to both moderate and high performing students taught with PBL, but did not make a compari-
son with matched non-PBL students. Geier et al. (2008) studied the effects of Center for Learning
Technologies in Urban Schools' (LeTUS) PBL curriculum with ~5,000 middle school science stu-
dents in Michigan, of which 91% were African American and 70% were low SES, from 1998 to
2001. They found significant gains on the Michigan Educational Assessment Program (MEAP) for
students participating in at least one of the LeTUS PBL units compared to the rest of the Detroit Pub-
lic School population. Schneider et al. (2002) also found that students taught through project-based
science (PBS) curriculum outperformed a matched national sample on 44% of tested items on the
National Assessment of Educational Progress' (NAEP) standardized science assessment.
1468
In the study, most closely paralleling ours, Holmes and Hwang (2016) studied the effects of PBL
instruction on eighth and ninth grade students' performance on standardized mathematics exams.
They compared results from a PBL school of similar size to the one in this study with those from a
larger conventional school in the same district. A notable strength of that study was the inclusion of
observations of mathematics instruction in both schools. However, the authors note that “the PBL
school had heavily recruited students from the existing high school in the same district,” (p.452).
Thus, although the PBL group and the comparison population were generally matched on student
characteristics, students were not randomly assigned to the two groups, and thus the study is subject
to the concern that students in the PBL group were not comparable to those at the conventional
school in terms of student and family interest and initiative. That study found no overall significant
difference between the PBL and the conventional group, but did see a narrowing of performance
gaps for some underrepresented groups, in particular the gap between low and higher-SES students
and between racial-ethnic minority and majority White students.
Prince and Felder (2006) make clear that many instructors actually employ a hybrid of project-
and problem-based teaching strategies, such that there is a continuum of implementation that these
authors characterize as PBL. Their meta-analysis found that students taught through PBL have
enhanced problem solving, communication and teamwork skills, and ability to apply content. Fur-
ther, they retain content knowledge longer than their traditional counterparts; PBL has a positive
impact on skill development when assessment was both given immediately and delayed.
Some studies find negative, or equivocal, effects, for the similar PrBL, particularly short-term
measures of surface, memorization-intensive knowledge or procedures (Prince & Felder, 2006). In a
meta-analysis, Dochy, Segers, Van den Bossche, and Gijbels (2003) found that students taught
through problem-based approaches have less surface-level content knowledge, but in exchange they
retain what knowledge they have gained longer. They found a significantly negative effect on knowl-
edge acquisition for students taught through PrBL, but did indicate that these findings were due to
two studies. Further, these differences in knowledge diminished after one to 2 years (Dochy
et al., 2003).
A later meta-synthesis of PrBL, a similar teaching methodology, done by Strobel and Barneveld
(2009) found similar results:
“Overall, students and staff indicated greater satisfaction with the [problem-based learn-
ing] approach to learning. Standardized tests that measured knowledge of basic science
focusing on short-term acquisition and retention (primarily the medical board exams in
their different versions) favored the conventional pedagogical approach across all stud-
ies. However, when the method used to assess basic science knowledge required a level
of elaboration beyond multiple-choice or true/false questions, results significantly
favored the PrBL [problem-based learning] approach. Standardized tests and other
assessment methods that evaluated skill-oriented application of knowledge, mixed
knowledge and long-term retention of knowledge, skills, and clinical performance sig-
nificantly favored PrBL” (p.54).
In summary, evidence indicates a strong effect of PBL on measures of deep and applied STEM learn-
ing, but not necessarily on immediate measures of knowledge of facts and procedures. Further, many
of the studies use convenience sampling or measure PBL occurring in restricted implementations.
Although some studies used questions similar to those found on high-stakes tests or old standardized
test questions released by the state for public access, most studies do not look at the effects of PBL
CRAIG AND MARSHALL
| 1469
on actual high-stakes test performance for randomized treatment groups. Geier et al. (2008) could be
said to have randomized treatment in that students were randomly assigned to the classrooms that
participated in the project-based curriculum and were compared to a group of students in the same
school district assigned to classrooms that did not; however, the study was conducted in a heavily
researcher-supported environment and results were only obtained in science classrooms at the middle
school level. In addition, the study took place in a larger district reform effort, which may have con-
tributed to the observed differences. This study has received criticism for not including a careful
comparison of the control and treatment characteristics (WWC, 2002).
We found no published empirical studies to date of the effects of an exclusively project-based
environment, as compared with conventional math and science instruction, on standardized test out-
comes in both math and science at the high-school level with randomized assignment of students to
control or treatment groups, thus controlling for self-selection bias. This study seeks to fill this gap.
7 | METHOD
7.1 | Site
7.1.1 | Manor independent school district
The central Texas school district used in this study, Manor Independent School District (MISD), cur-
rently serves 9,061 students. Manor proper is 6 miles east of Austin, Texas, but the district zoning
includes students from both east Austin and the more rural town of Manor. Until 2007, the district
provided only one high school for their students, Manor High School (MHS), which was utilized in
this study as the control group. In 2007, the district opened Manor New Technology High School
(MNTH), part of the New Tech Network, as a strategic initiative to improve school academic perfor-
mance, as designated by the Texas Education Agency (TEA), through providing an alternative learn-
ing environment for high school students.
Since opening in 2007, MNTH has received a lot of attention for their project-based school that
provides a 21st Century learning environment and unique school culture to the students in Manor.
Manor New Tech provides school tours to teachers and administrators around the world, and in 2013
President Barack Obama came for a visit and described MNTH as “an example for the Nation”
(KUT News, 2013). The district responded to the success of MNTH by starting district-wide PBL
initiatives. In 2010 the district created Think Forward, a 1 week PBL professional development train-
ing. Although open to other school districts, Manor ISD sent many of their own teachers (K-12) and
administrators through the Think Forward PBL training, in an attempt to spread the observed best
practices across the district. Since 2010, Manor ISD has redesigned and opened four project-based
elementary schools and one project-based middle school that are part of the New Tech Network, as
well as developed pathways for students in the district.
7.1.2 | Manor High School (control)

MHS is the main high school in the district, with a student population of 1,828. MHS consists of
90% historically underrepresented minority students where 71% of students face an economic disad-
vantage (Texas Tribune, 2019). In the years leading up to this study, MHS fluctuated between the
TEA designated grade of “academically acceptable” and “academically unacceptable” (Manor ISD,
2012). From 2003 to 2011, TEA used a state mandated accountability system to provide a school
grade to Texas schools of Exemplary, Recognized, Academically Acceptable, or Academically
1470
Unacceptable. These designations were measured by the following criteria for Texas schools:
(a) achievement on the Texas Assessment of Knowledge and Skills (TAKS) which is also the instru-
ment used in this study; (b) attendance rates; (c) high school completion rates; (d) annual dropout
rates for grades 7–12; and (e) college readiness (Texas Education Agency, 2019).
Due to the lower school performance rankings at MHS during the span of this research, the
instructional goals for the school were focused on whole class (Tier I) interventions for core concepts
and TAKS remediation, especially in the math and science classrooms. MHS had 10% of students
performing at or above grade level, which resulted in regular small group and individual pullouts and
other intervention methods being utilized. The district science and math curriculum initiatives aspired
to incorporate concrete, hands on curriculum practices, but achievement factors as well as experi-
enced teacher recruitment and retention concerns resulted in teacher development focused on class-
room management techniques and conventional teaching strategies (MISD, 2012). After the
completion of this study, strategic initiatives to implement PBL in all schools across the district were
implemented, and many MHS teachers were trained on PBL curriculum development and introduced
these best practices in their classes, but during the time of the study, project based learning was iso-
lated to Manor New Tech High.
7.1.3 | Manor New Tech High School (PBL)

Manor New Tech High School opened their doors in 2007 looking much more like a tech company
than a public high school. The classrooms all have glass walls and tables instead of desks and are
filled with top technology (e.g., SmartBoards, iMacs, Dell computers). From the very beginning,
Teachers at MNTH prided themselves on teaching all classes through project-based curriculum. Even
health and physical education classes used PBL. Every component of the school's unique design was
formulated to meet the needs of 21st century students, to prepare them for the careers of tomorrow.
Before opening the school all teachers and administrators were required to attend a two-week “New
Schools” training provided by the New Tech Network. During this training teachers and administra-
tors designed the learning outcomes (work ethic, collaboration, oral and written communication,
numeracy, and global and community engagement) that were incorporated into their project curricu-
lum in addition to state mandated content standards. All teachers were trained on how to create and
facilitate authentic, engaging projects that meet the criteria of The Buck Institute for Education's defi-
nition of PBL (Lamar & Ross). In 2009, MNTH was named as one of New Tech Network's flagship
campuses.
In addition to the requirement of PBL in the New Tech Network model, New Tech schools inte-
grate classes (i.e., History and English, Physics and Algebra), teach 21st century skills (i.e., learning
outcomes), and incorporate effective learning technology. MNTH also follows a “small school”
design model, serving 340 students compared with 1,445 students at MHS. Manor New Tech's
unique school culture can be attributed to the juxtaposition of all these components, as well as, an
active teacher and student School Culture Committee that meets regularly to ensure the core values
of respect, integrity, responsibility, perseverance, and trust are encompassed by the entire school
community.
In order to attend Manor New Tech, eighth grade students in the district, or their parents, submit
an application that has the student's name, gender, and contact information. All forms submitted by
the deadline are placed in a random lottery for admission. There are no academic requirements for
admittance to the school, although the lottery does control for gender differences in the applicant
pool by choosing 50 girls and 50 boys for each entering freshman class.
CRAIG AND MARSHALL
| 1471
One case study conducted at MNTH reported that the school outperformed both MHS, the con-
ventional high school in the district, and state averages, in terms of percent of students meeting the
standard on the 2011–2012 math and science TAKS (Gourgey, Asiabanpour, Crawford, Grasso, &
Herbert, 2009). Gourgey et al. (2009) identified four overarching themes through interviews with stu-
dents and teachers regarding their experience at the site for this research: (a) student engagement;
(b) student agency; (c) support of teachers' work; and (d) teacher agency. Lynch et al. (2017) con-
ducted an instrumental case study at the site of the current study, MNTH, exploring “critical compo-
nents” of effective STEM school environments. Data were collected from focus groups and
interviews with teachers, students, parents, and administrators, as well as classroom observations
using both the reformed teaching observation protocol (RTOP) and the Lesson Flow Classroom
Observation Protocol (LFCOP) instruments. Observational results found that STEM classes incorpo-
rated and provided ample opportunities for students to engage in PBL as defined by the literature.
This case study acted as a validation step, authenticating the claim that the school only teaches
through project based learning.
Lynch et al. (2017) described one project observed in a Geometry classroom where students were
challenged to design a building provided specific parameters. Students worked in their groups to des-
ignate roles and expectations for their group work, based on a teacher-created rubric. Students were
later observed utilizing technology (i.e., Geometer's Sketchpad) to revise and better represent their
rough blue prints created on graph paper. At one point, students were invited to attend a trigonometry
refresher, if needed, to continue working on their designs. During this class observation, 6% of time
was completely teacher-centered, 10% was devoted to teacher and small group interactions, 19% was
individual student work, and finally, 63% of time was spent in cooperative, student-centered project
activities.
7.2 | Sample
In order to look at the potential effects of attending MNTH for the entire high school career, the
cohort of interest for this study were students that applied to MNTH in the spring of 2008 and were
set to graduate in May 2012. Each year when students apply to MNTH, their names are put in a hat,
the school administration picks one male, and one female until the freshmen class of 100 students is
filled. If a student is accepted, but decides not to enroll in MNTH or decides to leave the school after
enrolling in the school, at the time of the study, the school did not automatically transfer students
from the waitlist to MNTH. This explains any observed differences in sample size from year to year
in the data. Also, the samples of students varied over the cohort trajectory through high school due to
missing data. If TAKS data were missing for students at any point in the study, the data were still
retained for the years that the data were available. Similarly, some students that applied to attend
MNTH were missing data for prior achievement. These students were still included in the study,
when data were provided.
Student records were divided into those that were picked in the lottery and attended MNTH and
those that were waitlisted and instead attended the conventional high school in district, MHS. These
data were then matched with student characteristics from the public education information manage-
ment system (PEIMS) database maintained by the TEA and then de-identified. For each student char-
acteristic variables in PEIMS include: student ethnicity data, gender, TAKS scores, socioeconomic
status, learning disability, English language learning (ELL) status, and graduation completion (Texas
Education Agency, 2019). Many of these student characteristic variables were utilized as
1472
independent variables to control for differences between the students waitlisted and attending MNTH
and in order to identify any effects on achievement gaps.
7.3 | Instrument
The TAKS was the state-mandated standardized test in Texas from 2002 to 2011 and was used as the
measure for student math and science achievement in this study. The TAKS test was designed by the
TEA to measure good instructional practice and capture student learning content curriculum stan-
dards, named the Texas Essential Knowledge and Skills (TEKS). TAKS scores were also used by the
TEA to assess school performance, and Texas high school students could not graduate without suc-
cessfully passing all four exit-level exams (Science, Math, Social Studies, and ELA). One critique of
PBL is that the extra time needed to allow students to construct their own understanding cannot feasi-
bly meet the demands of the large quantity of content curriculum standards, especially in math. To
investigate this criticism, the TAKS score seemed the most appropriate instrument to use in this study
that was comparing instructional methods and student learning.
The math TAKS was organized around the 10 content objectives and the science TAKS exam
was organized around five content objectives (Table 1). He mathematics TAKS exam was adminis-
tered to all Texas high school students in 9th, 10th, and 11th (exit-level), whereas the science TAKS
exam was administered only in 10th and 11th (exit-level).
Both the science and mathematics TAKS exam primarily consist of multiple-choice questions
organized around these learning objectives, and both math and science standards require students to
apply the respective contents in “real-life contexts.” In reviewing released TAKS questions, it
appears that the math questions' context is superficial whereas science questions require an under-
standing of the context. For example, the released math question in Figure 3 introduces an algebraic
expression that is not an authentic approach to solving that problem. This arbitrary use of an alge-
braic expression to solve an unrealistic context problem often distracts the student from successfully
solving the problem (Walkington, Sherman, & Petrosino, 2012). Also, although a student may enjoy
playing the guitar, knowledge of the context of a guitar is not necessary for correctly answering this
question. Whereas in Figures 4 and 5, the science questions require an understanding of the firework
design context in order to successfully answer the science questions.
T A B L E 1 Texas assessment of knowledge and skills (TAKS) content design blueprint
Mathematics objectives Science objectives

(1) functional relationships (1) nature of science
(2) properties and attributes of functions (2) Organization of Living Systems
(3) linear functions (3) interdependence of organisms
(4) linear equations and inequalities (4) structures and properties of matter
(5) quadratic and other nonlinear functions (5) motion, forces, and energy
(6) geometric relationships and spatial reasoning
(7) 2D and 3D representations
(8) measurement
(9) Percents, proportions, probability, and statistics
(10) mathematical processes and tools
CRAIG AND MARSHALL
| 1473
F I G U R E 3 The guitar problem: An example math in a “real-life context”

TAKS question. Problem 23 from the released 2009 11th grade math TAKS test
http://tea.texas.gov/student.assessment/taks/released-tests/. TAKS, Texas
assessment of knowledge and skills
F I G U R E 4 The firework problem diagram (part 1): An example TAKS question applying science to a context.
Problems #17–20 from the released 2009 11th grade science TAKS test http://tea.texas.gov/student.assessment/taks/
released-tests/. TAKS, Texas assessment of knowledge and skills
7.4 | Preparation of data

All data sets were matched by unique student identification numbers and completely de-identi-
fied. Student ethnicity data were provided as a nominal variable, coded by TEA with each ethnic-
ity associated with a numerical value. In order to prepare for analysis, three dummy coded
1474
F I G U R E 5 The firework problem questions (part 2): An example TAKS question applying science to a context.
Problems #17–20 from the released 2009 11th grade science TAKS test http://tea.texas.gov/student.assessment/taks/
variables were created titled Asian, African American, and Hispanic from the ethnicity variable
using “white, not of Hispanic origin” as the baseline for comparison. The data set did not have
any students listed as American Indian or Alaskan Native, so a separate variable was not created
for this ethnicity. Similarly, the female variable was transformed from M = male and F = female
in the original dataset to 0 = male and 1 = female. The state of Texas uses students' eligibility for
free or reduced lunch as an indicator of the socioeconomic status of students' families. These data
were originally coded 1 = eligible for free meals, 2 = eligible for reduced-price meals, 9 = other
economic disadvantage, and 0 = not identified as economically disadvantaged. The Economic
Disadvantage data were transformed and combined as a dichotomous variable, with 0 = not eco-
nomically disadvantaged and 1 = economically disadvantaged (free/reduced/other).
The TEA converts a nonlinear raw score on the TAKS test to a scaled-score, based on item diffi-
culty; for these exams the scaled scores run between ~1,300 and 2,900, depending on the administra-
tion year. The math and science TAKS scaled scores were also mean centered to allow for a more
appropriate interpretation of the intercept when creating the models.
CRAIG AND MARSHALL
| 1475
7.5 | Analysis and controls

The dependent variables in this study are math achievement measures on the TAKS mathemat-
ics standardized exams in grades 9, 10, and 11 (exit-level) and the science achievement mea-
sures on the TAKS science exams conducted in grades 10 and 11 (exit-level). The
independent variable is attendance at MNTH while controlling for the following student char-
acteristics; prior eighth grade TAKS achievement, special education, gifted and talented, ELL,
economic disadvantage, and gender, when appropriate. Prior to analysis, Levene's Test for
equality of variance was used to test for any significant difference in variance between the
control group and treatment group, thus eliminating any concern from differing sample size of
the groups. In order to determine any significant differences between the groups, Analysis of
Variance (ANOVA) was used to identify significant differences on continuous student charac-
teristic variables and χ 2 for categorical characteristics. Both tests were conducted a priori to
identify characteristic predictor variables to include in the regression models to control for
observed variances.
7.6 | Math and science achievement models

Linear regression is appropriate when the dependent variable is a continuous variable and indepen-
dent, predictor variables can be either continuous (e.g., achievement measures) or categorical
(e.g., ethnicity, English-language learning status). Further, in order to use linear regression, the fol-
lowing assumptions regarding the data must be met: (a) independence of observations, (b) linear rela-
tionship between the dependent variable and each independent variable, (c) normality of the error
distribution, (d) data homoscedasticity, (e) data multicollinearity, and (f) no significant outliers in
data, no high leverage points, and no highly influential points.
Students' random assignment to the accepted and waitlisted group meets the assumption of inde-
pendence of observations (Assumption #1), but, further, a Levene's test was used to confirm constant
variance of the two groups. In order to test the second assumption regarding the linear relationship
between the variables, a scatterplot matrix plotting the dependent variables against each level of the
predictor variables was created to ensure noncurved, linear relationships. Normality (Assumption #3)
was determined by creating a Q–Q plot for each dependent variable, confirming that data follow the
normality line. To check for homoscedasticity (Assumption #4), a residual plot was created for each
regression model confirming a random scatter showing equal distribution of points above/below and
left/right of “zero.” For all models, we looked at the collinearity output and confirmed that all VIF
values were less than 10, thus determining absence of multicollinearity (Assumption #5). Any out-
liers above 3 SD from the mean were removed in SPSS from the samples before creating the regres-
sion models, meeting the sixth assumption.
In order to answer the research question regarding mathematics and science achievement on the
TAKS measure for students attending MNTH, multiple linear regression models were created for the
results of each of the three high school TAKS math exams and two high school TAKS science
exams. We identified any significant differences between the two samples on these student character-
istics (special education, gifted and talented, ELL status, economic disadvantage, ethnicity, gender,
and prior eighth grade math and science achievement) and included these in the linear models to con-
trol for this variability where necessary. For each TAKS exam, the first model only included the main
effect of attending MNTH and the second model included ethnicity and economic disadvantage. Note
that these additional models included these student characteristic predictors since they were identified
1476
a priori as significantly different for the students accepted to MNTH and those waitlisted. Gender
was not included, as it did not meet this criterion.
We then created linear regression model equations (see Equation 1) for the dependent vari-
ables where the coefficients of the predictor variables (βj) represent the expected change in the
dependent variable (Y) as the corresponding predictor variable (xj) changes, holding all other pre-
dictor variables constant. This allowed us to determine if there were any statistically significant
effects of attending MNTH, while controlling for necessary student characteristics on students'
performance on their high school math and science TAKS exams. A significant coefficient (βi)
indicates that the difference observed in the dependent variable (i.e., TAKS score) is not likely to
have occurred by chance.
Y i = β0 + β1 x1 + β2 x2 + :: + βi xi ð1Þ
After creating the regression models, interaction variables were constructed between each
level of the student characteristic variables (e.g., English language learner, economic disadvan-
tage, ethnicity, and gender) and attendance at MNTH. Including these interaction terms in the
model allowed identification of any possible differences in effects of the student characteristics
and attendance at MNTH on high school standardized test achievement. For each of the cre-
ated interaction terms we tested for significant changes in the dependent variables, differences
in standardized test scores. If significant interaction terms were identified, interaction effect
was depicted by calculating the predicted means for each combination of group by assigned
school.
8 | RESULTS
8.1 | Descriptive
Of the 127 students that applied to MNTH, 41 students were not accepted (32%) and enrolled
in their ninth grade year at the other school in district, whereas 86 of the students (68%) were
accepted and enrolled in the PBL school (MNTH). There were observed differences in student
characteristic make-up between the two groups, but the only significant differences were for
the economic disadvantage and ethnicity. (see Tables 2–3, and 4). The χ 2 Test for Homogene-
ity revealed that the 44% of students that were accepted to MNTH labeled as economically
disadvantaged was significantly lower than the 68% of students identified as economically dis-
advantaged that were waitlisted χ 2(1, N = 127) = 6.464, (p < .05). Significant differences on
levels of the ethnicity variable for students accepted to MNTH were also identified
χ 2(1, N = 127) = 18.073, (p < .05). We then ran an ANOVA to look at differences in demo-
graphic make up for the continuous, prior-achievement TAKS variables. The differences on the
8th-grade math (p = .064) and science TAKS (p = .054) were not significant. Note there were
also no difference for gender, special education status, gifted and talented status, ELL status or
prior achievement, therefore it was not necessary to control for these characteristics in the lin-
ear regression models.
We controlled for these demographic differences of ethnicity, and economic disadvantage
between the sample that attended MNTH and the control group that were waitlisted by creating mul-
tiple linear regression models of students' math and science TAKS scores that included only atten-
dance at MNTH in the first model, then controlled the ethnicity variable and economic disadvantage
CRAIG AND MARSHALL
| 1477
T A B L E 2 Accepted and waitlisted groups' student characteristics as a percentage of the total sample
Characteristic Accepted (n = 86) Waitlisted (n = 41) Total applied (n = 127)

Special education 0.07 0.17 0.10
Gifted & talented 0.07 0.02 0.06
English language learner 0.02 0.07 0.04
Low SES 0.44 0.68 0.52
Ethnicity
Asian 0.04 0.00 0.03
African American 0.18 0.48 0.27
Hispanic 0.43 0.43 0.43
White 0.31 0.07 0.24
Female 0.59 0.56 0.58
Note: Due to missing data for different time stamps and characteristics, the sample size may vary from table to table.
T A B L E 3 Difference in categorical student characteristics for accepted and waitlisted groups
Characteristic χ2 df Sig. Effect

Special education 3.080 1 .079
Gifted & talented 1.098 1 .295
English language learner 1.829 1 .176
Low SES* 6.464 1 .011 Φ = − 0.226
Ethnicity*** V = 0.377
Asian 1.947 1 .163
African American 12.580 1 .000
Hispanic 0.000 1 .986
White 9.341 1 .002
Female 0.117 1 .732
Note: Due to missing data for different time stamps and characteristics, the sample size may vary from table to table. Significant
characteristics followed by *p < .05; **p < .01; ***p < .001 Φ or Cramer's V measuring effect size, were provided for significant
characteristics.
T A B L E 4 Difference in student prior achievement on standardized math and science exams in 8th grade
Standardized exam Accepted (n = 85) Waitlisted (n = 35) Total applied (n = 120)

Mathematics TAKS 2,182 (24.0) 2,110 (21.7) 2,159 (18.0)
Science TAKS 2,136 (26.8) 2049 (26.5) 2,111 (2047)
Note: Due to missing data for different time stamps and characteristics, the sample size may vary from table to table. Means are
displayed first, rounded to the nearest integer, followed by standard errors rounded to one decimal place in parentheses.
Significant characteristics followed by *p < .05 **p < .01 ***p < .001.
variable in the second step of the model. Since creating a second model for each data set increases
the likelihood of Type I error, a Bonferroni adjusted alpha of .025 was used to test for significance in
each regression model.
1478
8.2 | Linear regression models-mathematics

8.2.1 | Ninth-grade math TAKS
The linear regression model (see Table 5) shows an increased 96 scale score difference for students
that attended MNTH compared to those students that were waitlisted, which has a significant p-value
(B = 95.792, t = 2.220, p < .025). Then in the full model that controls for ethnicity and economic
disadvantage, there was a 65 scale score increase for students that attended MNTH; this result is no
longer significant (B = 64.702, t = 1.363, p = .176).
The coefficient of multiple determination, R2, is a measure of how well a multiple linear regression
model fits the data. For this study, R2 is the proportion of variance observed in the math and science
achievement measure that is explained by the regression model over the total variation. Cohen's
f 2(see Equation 2) is a measure of effect size, often used in multiple linear regressions. Cohen defined
f 2 values of .02 as a small effect size, f 2 = .15 as medium, and f 2 = 0.35 as large (Cohen, 1988).
Cohen's f 2 for the first ninth grade math TAKS achievement model is .048 which is a small effect size.
R2
Cohen0 s f 2 = ð2Þ
1 −R2
8.2.2 | Tenth grade math TAKS results

Although the linear regression model (see Table 6) shows an approximately 48 scale score increase
for students that attended MNTH, this difference is not significant (B = 48.019, t = 1.200, p = .233).
The full model that controls for ethnicity and economic disadvantage, is a scale score difference of
18 for students that attended MNTH, but still these differences are not significant (B = 17.925,
t = 409, p = .684).
8.2.3 | Eleventh grade math TAKS results

In the first linear regression model, although the difference observed was not significant, students
that attended MNTH had a scaled score increase of 60 on the 11th grade (exit-level) TAKS measure
T A B L E 5 Regression coefficients and standard errors for ninth grade student achievement mathematics measure
(TAKS) for attending MNTH (PBL) v. waitlisted (conventional instruction)
Model 1 Model 2
Variable B SEB β B SEB β

Constant 2071.696 38.124 2,160.964 60.039
Attended MNTH 95.792* 43.140 .214 64.702 47.471 .114
Low SES −35.949 38.373 −.097
Ethnicity
Asian 16.334 110.920 .015
African American −57.587 54.954 −.137
Hispanic −72.936 46.614 −.195
Note: Prior Achievement is centered at the mean. Each ethnicity variable was dummy coded compared to the baseline white student
population. B = unstandardized regression coefficient; SEB = standard error of the coefficient; β = standardized coefficient.
*p < .025; **p < .005. (Bonferroni adjusted α).
CRAIG AND MARSHALL
| 1479
T A B L E 6 Regression coefficients and standard errors for 10th grade student achievement mathematics measure
Model 1 Model 2

Constant 2,107.107 34.370 2,160.158 55.865
Attended MNTH 48.019 40.000 .116 17.925 43.844 .043
Low SES 21.373 38.168 .059
Ethnicity
Asian 43.584 111.012 .040
Hispanic −37.373 46.579 −.102
T A B L E 7 Regression coefficients and standard errors for 11th grade student achievement mathematics measure
Model 1 Model 2

Constant 2,200.880 39.304 2,188.107 64.402
Attended MNTH 60.320 45.788 .135 55.166 50.047 .124
Low SES −11.623 44.184 −.030
Ethnicity
Asian 106.227 147.247 .078
African American 2.333 63.543 .005
Hispanic 43.178 54.850 .110
(B = 60.320, t = 1.317, p = .191). Then the full model that controlled for ethnicity and economic dis-
advantage found that the difference in scaled score for students that attended MNTH decreased by
about five and the difference was not significant (B = 55.166, t = 1.102, p = .273) (see Table 7).
8.3 | Linear regression models-science

8.3.1 | Tenth grade science TAKS results
Both models for the 10th grade science TAKS exam found a significant effect of attending MNTH
on students' scores (p < .025) (see Table 8). The first linear regression model shows that attending
MNTH is associated with a 140 scale score increase over students that were waitlisted for MNTH,
and the difference is significant (B = 140.269, t = 3.637, p < .025). When economic disadvantage
and ethnicity are included in the model, the science TAKS measure scale score increased by 133 for
1480
T A B L E 8 Regression coefficients and standard errors for10th grade student achievement science measure (TAKS)
for attending MNTH (PBL) v. waitlisted (conventional instruction)
Model 1 Model 2

Constant 2097.000 33.083 2,113.945 54.660
Attended MNTH 140.269** 38.567 .336 133.082** 42.904 .319
Low SES 8.047 37.403 .022
Ethnicity
Asian 61.973 108.369 .056
Hispanic −28.633 45.598 −.077
students that attended MNTH and the difference was still significant (B = 133.082,
t = 3.102, p < .025).
The effect size was again calculated for both significant regression models for 10th grade science
TAKS achievement. For the initial regression model, comparing those that attended MNTH with
those that were waitlisted without controlling for any student characteristics, was (f2 = 0.127). When
the full model was created, controlling for economic disadvantage and ethnicity was (f 2 = 0.138).
According to Cohen (1988), both of these effect sizes for these models are close to a medium
effect size.
8.3.2 | Eleventh grade science TAKS results

The models developed showed a 76 scale score increase for students that attended MNTH, but this
difference was not significant (B = 76.194, t = 1.525, p = .131) (see Table 9). When economic dis-
advantage and ethnicity are included, the difference in scaled score is 133 for the treatment PBL
group attending MNTH, but still these differences are not significant (B = 133.082, t = 1.315,
p = 0.192).
8.3.3 | Achievement gap results

In order to fully answer the research question, we created interaction variables between the
accepted and attended MNTH variable and each of the following demographic characteristics:
English language learner, economic disadvantage, ethnicity, and gender. Twenty additional linear
regression models including the appropriate interaction variable were created to identify whether
any of the achievement gaps for the 9th, 10th, and 11th grade Math TAKS exams, as well as, the
10th and 11th grade Science TAKS exams increased, decreased, or were maintained when stu-
dents had attended MNTH. In the 20 linear regression models, there was not a single significant
interaction.
CRAIG AND MARSHALL
| 1481
T A B L E 9 Regression coefficients and standard errors for 11th grade student achievement science measure
Model 1 Model 2

Constant 2,196.720 42.883 2,113.945 54.660
Attended MNTH 76.194 49.957 .156 133.082 42.904 .319
Low SES 8.047 37.403 .022
Ethnicity
Asian 61.973 108.369 .056
Hispanic −28.633 45.598 −.077
9 | DISCUSSION
Our study found increased science achievement outcomes at the 10th grade level, no differ-
ence in science achievement in the 11th grade level, and no difference in mathematics
achievement at the 9th through 11th grade level for students engaged in PBL classrooms for
the entire duration of their high school career, compared with a control group of students that
chose to apply to MNTH and were waitlisted (see Table 10). The ninth grade math achieve-
ment model did find a significant difference in scaled scores of ~96, but this model did not
control for economic disadvantage or ethnicity and had a small effect size according to the
scale put forth by Cohen (1988).
Despite showing higher TAKS scores for students that attended the project-based MNTH in
all cases, none of the linear regression models with the SES and ethnicity controls created
showed a significantly different TAKS score, except for the 10th grade science TAKS test. Spe-
cifically, students attending MNTH scored an average increase of 119 scaled score on the 10th
grade science TAKS than their matched counterparts (B = 119.281, t = 3.012, p < .05). Since
data were provided as a raw-score and converted by TEA as a scaled-score, there is no way to
know the specific skills that the treatment group possesses that the control group lacks. Although
we can never know the exact, specific skills gained by experiencing math and science instruction
strictly in a PBL environment, the TEA does define cutoff of converted, scale-scores of greater
than 2,100 as “students met standard” for the content achievement exam. Since these tests are
high stakes, this means that performing below standard results in a requirement to repeat a grade
level or, for exit-level 11th grade exams, students must meet the standard scaled-score of 2,100
in order to graduate high school.
Finding significant gains in standardized test achievement on the science TAKS aligns with previ-
ous results. Geier et al. (2008) found that overall, students engaged in the project-based LeTUS sci-
ence curriculum in either cohort outperformed the rest of the Detroit public school population on the
state-mandated science MEAP standardized exam. Moreover, when the data were desegregated by
exposure to PBL (i.e., 7th grade only, 8th grade only, 7th and 8th grade), students that experienced
1482
T A B L E 1 0 Summary of linear regression results for students taught through PBL over conventional instruction
Model 1 PBL v. Conventional Model 2 PBL v. Conventional

(without controls) (with controls)
Measure B SEB β B SEB β

9th grade math TAKS 95.792* 43.140 .214 64.702 47.471 .114
10th grade math TAKS 48.019 40.000 .116 17.925 43.844 .043
11th grade math TAKS 60.320 45.788 .135 55.166 50.047 .124
10th grade science TAKS 140.269** 38.567 .336 133.082** 42.904 .319
11th grade science TAKS 76.194 49.957 .156 133.082 42.904 .319
Abbreviations: B, unstandardized regression coefficient; SEB, standard error of the coefficient; β, standardized coefficient.
*p < .05. **p < .01; *p < .025; **p < .005. (Bonferroni adjusted α).
PBL had significantly higher scores on the state-mandated science standardized exam than the non-
PBL group, except for one group of 7th graders in one cohort that saw no significant difference when
compared to the control group. Similarly, Schneider et al. (2002) found that students taught through
project-based science (PBS) curriculum outperformed the national average performance on the
NAEP standardized exam. The design of our study being different from the studies discussed above
could explain differences we found in our results.
There is less evidence for the effect of PBL on mathematics achievement. In a related field of eco-
nomics, Mergendoller et al. (2006) found results similar to the current study, that is, students taught
through problem-based methods (PrBL) performed as well as their traditionally taught counterparts
on microeconomics exams. Boaler (2002) did find significant increases in math performance on a
standardized math exam for students taught through PBL in England, but student admission to each
school was not discussed and therefore assumed not to have random PBL assignment, as is the larg-
est contribution of our study. Han et al. (2015) also found a positive influence of increasing exposure
to PBL on high school mathematics learning outcomes, but did not have a matched comparison
group.
Although there were no significant gains in mathematics achievement, the finding that attending
MNTH did not adversely affect achievement addresses the concern that PBL results in a lack of mas-
tery of fundamental skills and concepts (BIE, 2009), which standardized exams are designed to mea-
sure. Holmes and Hwang (2016) found no overall effect for PBL at the ninth-grade level, but did
find significant differences favoring PBL over conventional instruction on student achievement when
they disaggregated their data by SES and ethnicity; in contrast, we find an overall positive effect for
PBL that persists when there are no controls, but becomes insignificant when we control for ethnicity
and SES. The difference may lie in the fact that their study was not designed “to compare the PBL
and conventional approaches to teaching mathematics, nor was it designed to parse out a cause–effect
relationship between types of pedagogical approaches and learning outcomes” (Holmes & Hwang,
2016, p.455), whereas our study was expressly designed to compare PBL and the conventional
approach to teaching and learning. Further, the ability of our study to show significant results was
limited by the relatively smaller sample size.
Our results align with a less dramatic effect of PBL on mathematics achievement than science. It
may be that mathematics as it is typically taught in schools and assessed on standardized achieve-
ment exams, with the focus on memorizing algorithms and replicating computations, is less amenable
to the effects of contextualized learning. This may be particularly true for mathematics achievement
assessed in the Texas standardized tests, as discussed below.
CRAIG AND MARSHALL
| 1483
F I G U R E 6 The motor problem: An example TAKS question

applying science to a context. Problem 28 from the Released 2009 11th
grade Science TAKS test http://tea.texas.gov/student.assessment/taks/
F I G U R E 7 The hamburger problem: An example math in a “real-

life context” TAKS question. Problem 5 from the released 2009 11th
grade math TAKS test http://tea.texas.gov/student.assessment/taks/
One possible explanation for the significant increase in science TAKS score in 10th grade could
be that the type of story problems given on the science TAKS often require an application of the con-
tent to a context in order to successfully solve them. For example, the released science TAKS ques-
tion below is a question assessing students' understanding of the fifth science objective, Motion,
Forces, and Energy, and is situated in the context of a motor (see Figure 6). In order to successfully
answer this question, students will need at least a rudimentary understanding of the basic properties
of a motor, and further need to apply their understanding of motion, forces, and energy to this motor
contexts.
Math TAKS questions also often provide “real-life contexts” but after reviewing released TAKS
exams, these questions are not dependent on applying math to the context as is required in authentic
projects (PBL) and on many of the released science TAKS questions (see Figure 7). Perhaps this
explains the observed difference in effect of PBL between math and science.
This study was not designed to measure all benefits of PBL. It has previously been established
that students engaged in project-based curriculum develop critical thinking, collaboration, creativity
and research skills needed in the 21st century (Bell, 2010). Further, students engaged in projects
understand the concepts deeply and are able to retain the knowledge longer (Prince & Felder, 2006),
results that would not likely be captured in the standardized tests used to measure science and math
achievement.
9.1 | Implications and further research

Literature on PBL approaches suggests that students taught through these methods outperform stu-
dents on noncontent specific skills (e.g., task completion skills, problem-solving skills, collaboration
skills) (Cognition and Technology at Vanderbilt, 1992; Petrosino, 1998; Schmidt, Vermeulen, &
Van Der Molen, 2006; Yilmaz, Ren, Custer, & Coleman, 2010). In addition, students taught through
1484
F I G U R E 8 The Australia problem diagram: An example applying math to a “real-life context” PISA question.
Problem 148 from the released 2006 math PISA test https://www.oecd.org/pisa/38709418.pdf
project-based instruction show an increased interest in and motivation to pursue STEM fields (Pryor
et al., 1996; Tate, 1998; Verma, Dickerson, & McKinney, 2011; Yilmaz et al., 2010). Given that PBI
may yield these other benefits, our results showing that full-curriculum PBL is at least as effective as
conventional learning in science and math at the high school level on high-stakes outcomes imply
that PBL is a valuable option, at least for students who are open to alternative schooling, and merits
a place in school systems.
CRAIG AND MARSHALL
| 1485
Another implication may be the need for instruments to measure high-order thinking skills and
other non-content-specific skills (e.g., oral communication, technology literacy, critical thinking) as
applied in science and mathematics. The TAKS standardized math and science exams are predomi-
nantly multiple-choice exams and may not be sensitive to some higher-order skill development that
may occur when students are taught through project-based methods. On international tests, such as
the PISA, students in the U.S. are lagging behind their international counterparts on performance in
STEM (Casey, 2012). The mathematics PISA is different from traditional U.S. standardized achieve-
ment tests in that it is graded on a rubric, similar to American English achievement tests, allowing
for more open-ended questions and student responses. In Figures 8 and 9 the released PISA math
question requires students to apply their understanding of scale and surface area to approximate the
area of Australia. Comparing students that apply and attend PBL schools with students that are
waitlisted on PISA or on new high school success outcome measures that capture more robust learn-
ing than traditional U.S. standardized tests would permit further determination of the full effect
of PBL.
Another direction for future research would be an assessment of outcomes beyond high school for
students undergoing PBL. Although quantitative studies of college enrollment, degree attainment
and employment in STEM would be valuable, it would also be interesting to capture qualitative data
on students' perceived college and career preparedness, perception of STEM, and interest in STEM
through surveys and interviews.
10 | CONCL USI ON S AND L I M I T A T I O N S
There is a big push for innovative school reform in the current math and science education climate.
Charter schools and alternative schooling options that focus on science, technology, engineering, and
mathematics are opening nationwide in hopes of helping prepare the future generation for the 21st
century workplace. There is often concern that these inquiry-based, project-based methods are ideal,
but are not feasible with the time constraints and broad (mile-wide, inch-deep) content standards.
Overall results from this research showed that students that attended the project-based school per-
formed as well or significantly better on high school math and science TAKS than those that were
waitlisted and instead attended the traditional school, while controlling for ethnicity, SES, and prior
achievement.
The lack of significant improvements in mathematics could indicate that project-based learning
has a limited effect on students' high school outcomes, particularly in mathematics. Since the TEA
does not release item difficulty or student data disaggregated by objective, there is no way to deter-
mine the exact objectives or types of questions that the project based MNTH students were success-
ful on compared to the conventional MHS. Another explanation for this finding could be that PBL is
actually not occurring at the MNTH, but two separate case studies conducted at MNTH, however,
reported that authentic PBL is taking place in both STEM and non-STEM courses at MNTH
(Gourgey et al., 2009; Lynch et al., 2017). Perhaps the authentic implementation of PBL was not
consistent over time, but again, MNTH is designated a flagship campus by the New Tech Network
and thus visited randomly throughout the school year by politicians, administrators, and teachers
from around the globe, so it is very unlikely anything other than authentic PBL is typically occurring
at the campus. Another limitation between the two schools is that MNTH is a small school model
(less than 400 students) compared to the conventional school, MHS, that is sitting at 1,882. This
could explain the differences observed in the study that are attributed to PBL rather than school size.
1486
The relativity small size of the sample in this study may have also limited its abilty to produce
significant results. Further study with larger populations randomly assigned to PBL or conventional
instruction are warranted. Finally, it could be that unobservable characteristics, such as parental sup-
port and student motivation, have the largest effect on students' standardized test achievement,
regardless of the type of instruction students’ experience, limiting the generalizability of these results
to students like the population studied here. Despite these limitations, however, this study represents
a major contribution to the empirical literature on the effectiveness of PBL since it reports the first
randomized control study of the effect of PBL on high-stakes outcomes in mathematics and science
at the high-school level. Given the perceived urgency of improving the performance of US students
in STEM, particularly at the high school level, and the intensity of efforts to reform STEM education,
reform measures are likely to be adopted (as well as discarded) when they do not provide immediate,
positive results on science and math achievement, as measured by standardized assessments. Finding
that PBL may not be a better mechanism to increase standardized test scores, provides an important
benchmark against which to evaluate policy decisions in regard to PBL. Specifically, that standard-
ized achievement tests may not be the best measure of the effectiveness of PBL reform efforts.
O R C ID
Tara T. Craig https://orcid.org/0000-0003-3410-7474
R E F E REN CE S
Barron, B. J. S., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., & Bransford, J. D. (1998). Doing with
understanding: Lessons from research on problem- and project-based learning. Journal of the Learning Sciences, 7
(3–4), 271–311. https://doi.org/10.1080/10508406.1998.9672056
Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House: A Journal of
Educational Strategies, Issues and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415
Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-
based learning: Sustaining the doing. Supporting the Learning. Educational Psychologist, 26(3/4), 369–398.
Boaler, J. (1998). Open and closed mathematics: Student experiences and understandings. Journal for Research in
Mathematics Education, 29(1), 41–62. https://doi.org/10.2307/749717
Boaler, J. (2002). Paying the Price for “sugar and spice”: Shifting the analytical lens in equity research. Mathematical
Thinking and Learning, 4(2–3), 127–144. https://doi.org/10.1207/S15327833MTL04023_3
Bransford, J. D., Sherwood, R. D., Hasselbring, T. S., & Williams, S. M. (1990). Anchored instruction: Why we need
it and how technology can help. In Cognition, education, and multimedia: Exploring ideas in high technology.
Hillsdale N.J.: L. Erlbaum. (pp. 115–141).
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher,
18(1), 32–42. https://doi.org/10.3102/0013189X018001032
Buck Institute for Education (2019). Retrieved from Buck Institute for Education PBLWorks: http://pblworks.org.
Casey, B. (2012). STEM education: Preparing for the jobs of the future. Washington, DC: US Congress Joint Eco-
nomic Committee.
Cognition, & Technology Group at Vanderbilt. (1992). The Jasper series as an example of anchored instruction: The-
ory, program description, and assessment data. Educational Psychologist, 27(3), 291–315. https://doi.org/10.1207/
s15326985ep2703_3
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, N.J: Routledge.
Cullen, J. B., Jacob, B. A., & Levitt, S. D. (2005). The impact of school choice on student outcomes: An analysis of
the Chicago public schools. Journal of Public Economics, 89(5–6), 729–760. https://doi.org/10.1016/j.jpubeco.
2004.05.001
Dewey, J. (1938). Experience and education. Carbondale: Southern Illinois University Press.
CRAIG AND MARSHALL
| 1487
Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis.
Learning and Instruction, 13(5), 533–568. https://doi.org/10.1016/S0959-4752(02)00025-7
Geier, R., Blumenfeld, P. C., Marx, R. W., Krajcik, J. S., Fishman, B., Soloway, E., & Clay-Chambers, J. (2008). Stan-
dardized test outcomes for students engaged in inquiry-based science curricula in the context of urban reform. Jour-
nal of Research in Science Teaching, 45(8), 922–939. https://doi.org/10.1002/tea.20248
Gourgey, H., Asiabanpour, B., Crawford, R., Grasso, A., & Herbert, K. (2009). Case study of manor new tech high
school: promising practices for comprehensive high schools. Retrieved from http://www.newtechnetwork.org/sites/
default/files/dr/manornewtechcasestudy.pdf
Han, S., Capraro, R., & Capraro, M. M. (2015). How science, technology, engineering, and mathematics (STEM)
project-based learning (PBL) affects high, middle, and low achievers differently: The impact of student factors on
achievement. International Journal of Science and Mathematics Education, 13(5), 1089–1113. https://doi.org/10.
1007/s10763-014-9526-0
Hofstein, A., & Lunetta, V. N. (2004). The laboratory in science education: Foundations for the twenty-first century.
Science Education, 88(1), 28–54.
Holmes, V.-L., & Hwang, Y. (2016). Exploring the effects of project-based learning in secondary mathematics educa-
tion. Journal of Educational Research, 109(5), 449–463.
Kilpatrick, W. (1918). The project method. The Teachers College Record, 19(4), 319–335.
Kliebard, H. M. (2004). The struggle for the American curriculum (pp. 1893–1958). New York, NY: Psychology
Press.
Knoll, M. (1997). The project method: Its vocational education origin and international development. Journal of Indus-
trial Teacher Education, 34, 59–80.
Krajcik, J., & Blumenfeld, P. C. (2006). Project-based learning. In The Cambridge handbook of the learning sciences
(pp. 317–333). New York: Cambridge Press.
Larmer, J., & Ross, D. (2009). PBL Starter Kit. Retrieved October 25, 2018, from http://www.bie.org/object/books_
others/pbl_starter_kit_2nd_ed
Lynch, S. J., Spillane, N., House, A., Peters-Burton, E., Behrend, T., Ross, K. M., & Han, E. M. (2017). A policy-
relevant instrumental case study of an inclusive STEM-focused high school: Manor new tech high. International
Journal of Education In Mathematics Science and Technology, 5(1), 1–20. https://doi.org/10.18404/ijemst.75656.
https://doi.org/10.18404/ijemst.75656
Manor Independent School District (2012). Manor ISD district strategic improvement plan. Retrieved from http://
www.manorisd.net.
Mergendoller, J., Maxwell, N., & Bellisimo, Y. (2006). The effectiveness of problem-based instruction: A comparative
study of instructional methods and student characteristics. Interdisciplinary Journal of Problem-Based Learning, 1
(2), 49–69. https://doi.org/10.7771/1541-5015.1026
Mills, J. E., & Treagust, D. F. (2003). Engineering education—Is problem-based or project-based learning the answer?
Australasian Journal of Engineering Education, 3, 2–16.
Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds of knowledge for teaching: Using a qualitative
approach to connect homes and classrooms. Theory Into Practice, 31(2), 132–141. https://doi.org/10.1080/
00405849209543534
Murnane, R. J., Newstead, S., & Olsen, R. J. (1985). Comparing public and private schools: The puzzling role of selec-
tivity bias. Journal of Business & Economic Statistics, 3(1), 23–35. https://doi.org/10.2307/1391686
New Tech High Foundation. (2010). Retrieved from The New Tech Network website: http://www.newtechnetwork.org
Petrosino, A. J. (1998). At-risk children's use of reflection and revision in hands-on experimental activities. (ProQuest
dissertations and theses; Thesis PhD). Vanderbilt University; Publication Number: AAI9827617; ISBN:
9780591798975; Source: Dissertation Abstracts International, Volume: 59-03, Section: A, Page: 0725.;
248 P. Retrieved from http://adsabs.harvard.edu/abs/1998PhDT.......171P
OECD. (2012). PISA 2012 Results: What Students Know and Can Do: student performance in mathematics, reading,
and science. Volume I. Paris, France: OECD.
Prince, M. J., & Felder, R. M. (2006). Inductive teaching and learning methods: Definitions, comparisons, and research
bases. Journal of Engineering Education, 95(2), 123–138.
Pryor, A. Z., Conger, J. E., Galardi, R. A., Hetter, K. B., Huebel-Drake, M., Mouradian, M. E., … & Stern,
E. L. (1996). Putting trust in change: A partnership for better science education. T.H.E. Journal, 24(3), 74–76.
1488
Rivet, A. E., & Krajcik, J. S. (2004). Achieving standards in urban systemic reform: An example of a sixth grade
project-based science curriculum. Journal of Research in Science Teaching, 41(7), 669–692. https://doi.org/10.
1002/tea.20021
Schmidt, H. G., Vermeulen, L., & Van Der Molen, H. T. (2006). Longterm effects of problem-based learning: A com-
parison of competencies acquired by graduates of a problem-based and a conventional medical school. Medical
Education, 40(6), 562–567. https://doi.org/10.1111/j.1365-2929.2006.02483.x
Schneider, R. M., Krajcik, J., Marx, R. W., & Soloway, E. (2002). Performance of students in project-based science
classrooms on a national measure of science achievement. Journal of Research in Science Teaching, 39(5),
410–422. https://doi.org/10.1002/tea.10029
Strobel, J., & van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing
PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1), 44–58. https://doi.
org/10.7771/1541-5015.1046
Tate, R. L. (1998). An introduction to modeling outcomes in the behavioral and social sciences (2nd ed.). Edina, MN:
Burgess International Group, Inc.
Texas Education Agency (2013). Retrieved from Texas Education Agency: https://tea.texas.gov
Texas Education Agency. (2019). Retrieved from the Texas Education Agency website: http://www.tea.texas.gov
Texas Tribune. (2019). Retrieved from the Texas Tribune Public Schools Explorer website: http://www.schools.
texastribune.org
Thomas, J. W. (2000). A review of research on project-based learning.
Verma, A. K., Dickerson, D., & McKinney, S. (2011). Engaging students in STEM careers with project-based
learning--MarineTech project. Technology and Engineering Teacher, 71(1), 25–31.
Walkington, C., Sherman, M., & Petrosino, A. (2012). “Playing the game” of story problems: Coordinating situation-
based reasoning with algebraic representation. The Journal of Mathematical Behavior, 31(2), 174–195. https://doi.
org/10.1016/j.jmathb.2011.12.009
What Works Clearinghouse (WWC). (2002). U.S. Department of Education's Institute of Education Sciences (IES).
Retrieved from https://ies.ed.gov/ncee/wwc/
Yilmaz, M., Ren, J., Custer, S., & Coleman, J. (2010). Hands-on summer camp to attract K -12 students to engineering
fields. IEEE Transactions on Education, 53(1), 144–151. https://doi.org/10.1109/TE.2009.2026366
How to cite this article: Craig TT, Marshall J. Effect of project-based learning on high
school students' state-mandated, standardized math and science exam performance. J Res Sci
Teach. 2019;56:1461–1488. https://doi.org/10.1002/tea.21582

Effect of Project-Based Learning On High School Students' State-Mandated, Standardized Math and Science Exam Performance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Effect of Project-Based Learning On High School Students' State-Mandated, Standardized Math and Science Exam Performance

Uploaded by

Copyright:

Available Formats

Received: 10 January 2018 Revised: 8 May 2019 Accepted: 1 June 2019

Effect of project-based learning on high school

Tara T. Craig1 | Jill Marshall2

grade science TAKS measure by (B = 133.082, t = 3.102,

5 | INDUCTIVE TEACHING METHODS

MATH & SCIENCE

F I G U R E 2 A schematic of the conceptual

6 | RESEARCH ON PBL AND STEM LEARNING

7.1.2 | Manor High School (control)

7.1.3 | Manor New Tech High School (PBL)

T A B L E 1 Texas assessment of knowledge and skills (TAKS) content design blueprint

Mathematics objectives Science objectives

F I G U R E 3 The guitar problem: An example math in a “real-life context”

7.4 | Preparation of data

7.5 | Analysis and controls

7.6 | Math and science achievement models

Characteristic Accepted (n = 86) Waitlisted (n = 41) Total applied (n = 127)

T A B L E 3 Difference in categorical student characteristics for accepted and waitlisted groups

Characteristic χ2 df Sig. Effect

Standardized exam Accepted (n = 85) Waitlisted (n = 35) Total applied (n = 120)

8.2 | Linear regression models-mathematics

8.2.2 | Tenth grade math TAKS results

8.2.3 | Eleventh grade math TAKS results

Variable B SEB β B SEB β

Variable B SEB β B SEB β

Variable B SEB β B SEB β

8.3 | Linear regression models-science

Variable B SEB β B SEB β

8.3.2 | Eleventh grade science TAKS results

8.3.3 | Achievement gap results

Variable B SEB β B SEB β

Model 1 PBL v. Conventional Model 2 PBL v. Conventional

Measure B SEB β B SEB β

F I G U R E 6 The motor problem: An example TAKS question

F I G U R E 7 The hamburger problem: An example math in a “real-

9.1 | Implications and further research

10 | CONCL USI ON S AND L I M I T A T I O N S

Tara T. Craig https://orcid.org/0000-0003-3410-7474

You might also like