Professional Documents
Culture Documents
Edd 1105 - Literature Review Revised 1
Edd 1105 - Literature Review Revised 1
Cristina Montagna
Abstract
Teaching (JiTT), which blends technology-based formative assessment with in-class teaching.
Current research evidence supports the effectiveness of JiTT on academic achievement in the
field of high school education, but similar research in higher education is limited. This literature
review analyzed recent studies which have investigated the use of JiTT in higher education, in
effort to identify how it supports: (a) academic achievement, (b) student academic perceptions,
and (c) student academic engagement. Within the review, strengths and limitations of the studies
are also discussed, as well as implications and suggestions for future research.
Problem Statement
transform traditional curricula to better prepare students for success in today’s global economy
(McLaughlin et al., 2014). These institutions are seeking instructional reform that is student-
centered and promotes increased use of technology, to better meet the diverse needs of students
assessment method known as Just-in-Time Teaching (JiTT), which is aligned with these needs.
Research on JiTT in higher education is limited and has mainly been conducted with
undergraduate students in math and science fields, but the existing research shows relatively
positive results (Chantoem & Rattanavich, 2016; De, Kavitha, & Kanagasabai, 2014; Liberatore,
Morris, & Vestal, 2017; Luo, 2018; Lopez-Cupita, 2015; Marrs & Novak, 2014; Natarajan &
Bennet, 2014; Osmond & Goodnough, 2012; Riskowski, 2015; Sayer, Marshman, & Singh,
2016; Schuller, DaRosa, & Crandall, 2015; Strayer, 2012; Sun, Martinez, & Seli, 2014; Talley &
Scherer, 2013; Wanner, 2016) (see Appendix). This literature review explored current research
on JiTT in higher education and what the findings of these studies may imply for supporting
student: (a) academic achievement, (b) academic perceptions, and (c) engagement.
questions, compared of various item types (e.g., multiple choice, true/false, and open ended
questions) on upcoming course material outside of class and submit their responses via an online
course management system (e.g., Blackboard), a few hours before each class begins (Riskowski,
2015). Students' responses to JiTT exercises are reviewed by the instructor and are used to
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 4
develop classroom learning activities addressing learning gaps. Thus, the identification and
JiTT is based on a feedback loop model between the web-based assessment questions
(outside the classroom) and the in-classroom learning (Figure 1) (Marrs & Novak, 2014).
Instructors use online course management systems to post course assessment questions before
class, and students use this web-based material to prepare for each class. In addition, student
responses to these questions allow the instructor to evaluate the students’ current understanding
of concepts or abilities, and these responses provide an opportunity for the instructor to adapt the
(Riskowski, 2015). Thus, the instructor uses student responses to create an interactive classroom
environment that emphasizes active learning and cooperative problem solving and decreases the
Figure 1. JiTT Feedback Loop Model which depicts the relationship between web-based
assessment questions (outside the classroom) and in-classroom learning. Adapted from "Just-in-
Time Teaching in Biology: Creating an Active Learner Classroom Using the Internet" by Marrs,
Summary of Literature
This section explored the extant literature pertaining to the use of JiTT in higher
education. This summary of the literature was guided by the following research question: In
what ways does utilizing JiTT, in college courses, support academic success? Review of the
related studies suggested that JiTT improves several factors related to academic success,
including: (a) academic achievement, (b) student academic perceptions, and (c) student academic
engagement (see Appendix). What follows is a review of studies related to each of the
aforementioned domains.
Academic Achievement
Several experimental (Chantoem & Rattanavich, 2016; Liberatore et al., 2017; Luo,
2018; Natarajan Bennet, 2014; Riskowski, 2015) and mixed-methods (Talley & Scherer, 2013)
studies have explored the relationship between JiTT and student academic success in higher
education. These studies have been conducted primarily in science and math courses, with
the purpose of investigating the effects of JiTT on student academic performance, across two
incorporated a: (a) treatment group, a course which received a lecture with web-based JiTT; and
(b) control group, a course which received only a traditional lecture. The results of the study
found that the treatment group had a significantly higher overall course grade (by one letter
grade on average) than the control group, thus demonstrating that JiTT improved students'
performance.
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 6
(2014), across two undergraduate Calculus courses at a Kansas university. This study differed
with Lou’s (2018) in that Natarajan and Bennet (2014) released the JiTT web-based assessments
a few weeks before their dues dates, rather than a few days prior to when the associated topics
would be covered in a lecture, the latter of which is a traditional JiTT procedure. The results of
their study not only indicated that JiTT improved students’ academic achievement, but also
revealed that timing of the JiTT web-based assessments was not a crucial factor when it came to
improving student achievement. The implication of the this study is that JiTT is a flexible tool
that instructors can easily adapt to address gaps in students’ background knowledge, without
having to follow a pure just-in-time approach, as gains in learning occurred regardless of the
Liberatore et al. (2017) also conducted a pretest/posttest experimental study across two
undergraduate Engineering courses at a Colorado university. Their treatment and control groups
were not unlike those developed by Lou (2018) and Natarajan and Bennet (2014).
The results of Liberatore et al.’s (2017) study showed that student performance covering a
specific course topic, reinforced by JiTT, was found to be measurably higher (>9%) than the
control group who did not receive the JiTT assessment. Unlike the previously discussed studies,
Liberatore et al.’s (2017) study also found higher median and 3rd quartile scores on posttests,
indicating that JiTT may be beneficial to mid-performing students. One possible explanation is
that these mid-performing students gained more benefit from repetition of challenging material
via JiTT than the high or low performing students (p. 10). The implication of this study for
educators is that if mid-performing students have the most to gain from JiTT, then this could
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 7
translate into higher overall course grades, again demonstrating that JiTT can improve students'
academic performance.
The studies reviewed thus far, focus on introductory undergraduate level courses.
Current research utilizing experimental design has rarely focused on upper-division courses
(Riskowski, 2015). Recognizing this gap in the literature, Riskowski (2015) designed an
pretest/posttest experimental study across two Physics courses, designed for seniors, at a
university in Texas. This study showed stronger academic achievement (e.g., 50% gain in
posttest scores) in the students who were in the JiTT course relative to the traditional lecture
course (p. 176). The results of this study suggested that JiTT assessments were effective in
other comparable studies have shown it to be in introductory level courses. Caution must still be
used in generalizing these findings to other populations as participants were primarily Hispanic
males (70%).
Similar achievement results were found in a study conducted outside of the United
States. Chantoem and Rattanavich (2016) conducted a pretest/posttest experimental study across
two English language courses at a vocational college in Thailand. The results of their study
found that the experimental group registered higher scores than the control group in English
reading and writing abilities, which further addresses the effectiveness of JiTT. But, unlike the
previous studies, the present study used a randomized true control group pretest/posttest design.
In this study, the randomized procedure was meant to ensure that marked differences between
the groups after the intervention were attributable to differences in received instruction, thus,
The fact that the studies developed by Lou (2018), Natarajan and Bennet (2014),
Liberatore et al. (2017), and Riskowski (2015) did not randomly assign students to either the
JiTT or traditional class, is a major limitation they all share. Because of the lack of random
assignment, generalizations to larger population groups cannot be made. Also, these studies, in
addition to Chantoem and Rattanavich’s (2016) research, were limited by several other factors,
including concerns that: (a) the studies were each discipline-specific (e.g., Geography, Calculus,
Engineering, Physics, or English language courses); (b) the studies were limited to
undergraduates; (c) in each study, the data was gathered from only one institution in only one
country; and (d) the researchers in each study were also the instructors in both the JiTT and
Scherer (2013) support the results of the previously discussed experimental studies. Tally and
undergraduate Psychology course at a mid-Atlantic university. Student course grades and survey
responses were compared for two courses, one of which was JiTT-supported. The results of the
study found student academic performance was significantly higher, in the JiTT-supported
course, than in the previous semester in which JiTT was not utilized. Despite providing support
for the effectiveness of JiTT on student achievement, this study is limited by the participants’
demographics (i.e., all African American), and an undisclosed sample size, which make the
Academic Perceptions
Several qualitative (Marrs & Novak, 2014; Osmond & Goodnough, 2012; Sayer et al.,
2016) and mixed-methods (De et al., 2014; Strayer, 2012) studies have explored the relationship
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 9
between JiTT and student academic perceptions in higher education. These studies have been
conducted in science and math courses, with results indicating that students perceive JiTT as
more supportive of their content learning and academic improvement, than traditional lecture-
only approaches.
Qualitative studies. Marrs and Novak (2014) designed a qualitative study, involving
750 students enrolled in undergraduate and graduate Biology courses at a university in Indiana.
The purpose of the study was to explore the students’ perspectives (via a self-report
questionnaire) regarding the effect of JiTT on their learning (in Biology classes). The results of
the study found that 94% of students reported that JiTT assessments were “very useful to
learning the fine points” of a concept (p. 57). In addition, these students reported an average
normalized gain of ~60% on test questions that were reinforced by JiTT assessments (p. 57).
A case study conducted by Osmond and Goodnough (2012) involved interviews with an
instructor, who taught two undergraduate Science courses (one that was JITT-supported and one
that was traditional lecture), at a U.S. university. The instructor similarly reported that students
who participated in her JiTT-supported class typically performed better academically than
students in her lecture only course. This study was important, as it explored instructor
perceptions, as opposed to student perceptions. The positive findings of both qualitative studies
are consistent with current quantitative research indicating that JiTT is effective in improving
The articles reviewed thus far, with regard to academic perceptions, focus on introductory
level undergraduate courses. As such, the use of JiTT with students in upper-division courses
has received less attention, particularly in qualitative research. Due to this gap in the literature,
Sayer et al. (2016) conducted a case study with the purpose of exploring the effects of JiTT on
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 10
student academic performance in an undergraduate Physics course, designed for juniors and
seniors, at a United States university. As in Marrs and Novak’s (2014) study, Sayer et al. (2016)
found similar results when comparing students responses on self-reported questionnaires. The
students attributed higher performance on test scores to the reinforcement provided by JiTT
assessments.
The findings discussed within these qualitative studies suggest that JiTT assessments
were effective in improving students' academic performance, in both introductory and upper-
division undergraduate courses. Despite this, the studies were limited by the fact that survey
data was collected, which is subject to response bias. In addition, all three studies lacked thick
qualitative description (of participant survey responses) and data triangulation methods (as only
survey data was collected), which are both hallmark to qualitative research (Heppner et al.,
2016).
medical students at a university in Malaysia, in effort to explore their perspectives on JiTT over
traditional lecture teaching. The results of their study highlighted that JiTT assessments were
perceived superior to traditional lecturing with statistically significant outcomes in the clarity of
the topic (p = 0.003) and knowledge gained for exams (p = 0.044) (p. 13). Once again, these
results support the acceptability of JiTT over traditional lecture teaching. Despite this strength,
limitations include the fact that survey data was collected, which is subject to response bias, and
the study was conducted at one university, outside of the United States, which limits the
generalizability of results.
In a U.S.- based mixed-methods study, Strayer (2012) explored student satisfaction with
two sections of an undergraduate Statistics course (one JiTT and one traditional), at a Midwest
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 11
university. All participants were interviewed and completed a survey. The study’s qualitative
and quantitative results found that students in the JiTT classroom preferred learning via
technological innovation (i.e., JiTT web-based assessments) when compared to the traditional
lecture class. A major implication of these results for educators is that students prefer blended
Strayer’s (2012) study was not without limitations involving the sample and potential
bias. In this study, the researcher was also the instructor in both the JiTT and traditional classes,
which makes results subject to researcher bias. As survey data was collected, it too is subject to
response bias. Other limitations included the sample size and the fact that students were not
randomly assigned to either the JiTT or traditional lecture classroom. Because of the lack of
Academic Engagement
One main concern with the traditional lecture format is that students’ levels of
engagement tend to be low, which may be correlated with reduced learning levels (Wanner,
2016). In recent years, technology has started to be applied in lecture halls and in online settings
(i.e., JiTT) to address this issue (Sun et al., 2014). For the purposes of this literature review,
Several quantitative (Schuller et al., 2015), qualitative (Lopez-Cupita, 2015; Wanner, 2016), and
quasi-experimental (Sun et al., 2014) studies have explored the relationship between JiTT and
student academic engagement in higher education. These studies have been conducted primarily
in physical and social science courses, with results supporting the positive influence of JiTT on
Quantitative study. The purpose of Schuller et al.’s (2015) quantitative study was to
explore the difference in learner engagement when JiTT was used in place of traditional
lectures. Students were randomly assigned to either a JiTT or lecture-only medical course, at an
Illinois university. Student in-class engagement was studied by video analysis conducted by two
reviewers. Video analysis revealed that participants were engaged 34% of the time during the
JiTT classes compared with 11% of the time during lecture only classes (p. 389). These results
indicate that students spent more time engaged in JiTT-supported classes as opposed to lecture-
only classes, thus supporting the effectiveness of JiTT at increasing student engagement levels
during in-class learning. Random assignment and high inter-rater reliability (ICC = 0.98)
employed by this study yields more convincing results, but the small sample size (N = 31) limits
Qualitative study. The results of Wanner’s (2016) qualitative study, supports Schuller et
al.’s (2015) findings. The purpose of Wanner’s (2016) study was to investigate undergraduate
students’ perceptions of the influence of JiTT assessments on their in-class engagement levels.
The interview-based study found that 70% of students agreed that JiTT assessments were helpful
for their understanding of the in-class lecture content, and thus, resulted in increased engagement
during in-class time (i.e., participation) (p. 160). Yet, the generalizability of this study is limited
by several factors: (a) survey data, which is subject to response bias; (b) location of study (i.e.,
Australia); and (c) a lack of data triangulation methods (as only survey data was collected).
through the JiTT strategy, students could organize their ideas before class and as a result they felt
more confident to participate actively during the in-class lecture, implying that engagement was
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 13
engendered before the lesson. A major strength of this study was the use of data triangulation
(via interviews and journal artifacts), yet a weakness was the use of a convenience sample, which
was unlikely to be representative of the population being studied, thus limiting its
generalizability.
studies investigating JiTT have reported the benefits of increased behavioral engagement
(Schuller et al., 2015; Wanner, 2016), but current studies have rarely examined the influence of
JiTT on other types of student engagement (i.e., emotional and cognitive). To address this gap in
the literature, Sun et al. (2014) devised a quasi-experimental study in which undergraduate
university in the southwest of the U.S. To measure levels of student engagement, all participants
completed the School Engagement Questionnaire (Fredricks, Blumenfeld, Friedel, & Paris,
2005), for which high internal consistency reliability was reported (α = .917). The results of the
study found that students in the JiTT-supported class had significantly higher levels of emotional
engagement, cognitive engagement, and overall engagement than those in lecture-only class (p.
240). These results suggest that the use of JiTT creates an environment that facilitates students’
positive emotions and helps students concentrate on the classroom instruction. Caution is
warranted in generalizing the results of this study to other populations, as participants were not
Discussion
The purpose of this literature review was to explore ways in which utilizing JiTT, in
college courses, supported academic success. Review of the extant literature revealed that JiTT
supported student academic performance with regard to three domains: (a) academic
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 14
achievement, (b) academic perceptions, and (c) engagement. Analysis of the results of the varied
To begin with, the results supported the positive influence of JiTT on student
achievement, especially for mid-performing students (Liberatore et al., 2017), regardless of the
timing of JiTT assessments (Natarajan & Bennet, 2014), and whether or not the course level was
introductory (Chantoem & Rattanavich, 2016; Liberatore et al., 2017; Luo, 2018; Natarajan &
Bennet, 2014; Talley & Scherer, 2013) or upper-division (Riskowski, 2015). Next, the results
indicated that students perceived JiTT as more supportive than traditional lecture methods of
their content learning and academic improvement, regardless of whether or not the course level
was introductory (De et al., 2014; Marrs & Novak, 2014; Osmond & Goodnough, 2012; Strayer,
2012) or upper-division (Sayer et al., 2016). Last, the results supported the positive influence of
2015; Schuller et al., 2015; Wanner, 2016), as well as social and emotional engagement during
Implications
The results of the researched studies supported JiTT’s influence on student academic
and teaching methodology in higher education. In addition, these robust studies varied in
methodology and geographical location, yet many of them: (a) utilized small sample sizes; (b)
pertained to primarily Math and Science courses; and (c) only one of the studies included
graduate-level students (see Appendix). Thus, as educational institutions are currently seeking
instructional reform that incorporates technology, in effort to change the method from traditional
lectures to JiTT in higher education, multi-course studies with larger and more diverse
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 15
populations need to be conducted to support it as a more feasible teaching and assessment tool
(De et al., 2014). In addition, educational outcomes need to be compared amongst these diverse
groups. Closely linked with this idea is the need for cross-cultural studies in effort to evaluate if
results are similar in other student populations, which will indicate whether or not such results
Of additional importance is the mention of the fact that review of the literature finds that
there is little to no research available on how JiTT influences instructor/teacher knowledge and
practice (Osmond & Goodnough, 2012). Thus, this topic presents a major gap in the current
literature, and another potential avenue for future research. Despite this, the nature of the JiTT
strategy lends itself to the enhancement of many areas of pedagogical content knowledge, as "the
strategy allows educators to probe students’ prior understandings, obtain feedback from students,
and foster engaging face-to-face learning" (Osmond & Goodnough, 2012, p. 78).
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 16
References
Chantoemi, R., & Rattanavich, S. (2016). Just-in-time teaching techniques through web
technologies for vocational students’ reading and writing abilities. English Language
De, S., Kavitha, N., & Kanagasabai, S. (2014). Acceptability of just-in-time teaching amongst
doi:10.5959/eimj.v6i1.186
Heppner, P. P., Wampold, B. E., & Owen, J. (2016). Research Design. Boston, MA: Cengage.
Liberatore, M. W., Morrish, R. M., & Vestal, C. R. (2017). Effectiveness of just in time teaching
doi:10.5408/1089-9995-56.2.166
Marrs, K. A., & Novak, G. (2014). Just-in-time teaching in biology: Creating an active learner
classroom using the internet. Cell Biology Education, 3(1), 049–061. doi:10.1187/cbe.03-
11-0022
McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A., Griffin, L.
M., Mumper, R. J. (2014). The flipped classroom: A course redesign to foster learning
and engagement in health professions school. Academic Medicine, 89(2), 236-243. doi:
10.1097/acm.0000000000000086
JITT'S INFLUENCE ON ACHIEVEMENT: A LITERATURE REVIEW 17
Natarajan, R., & Bennett, A. (2014). Improving student learning of calculus topics via modified
10.1080/10511970.2013.854853
Osmond, P., & Goodnough, K. (2011). Adopting just-in-time teaching in the context of an
77-91. doi:10.1080/17425964.2011.558387
Sayer, R., Marshman, E., & Singh, C. (2016). Case study evaluating just-in-time teaching and
peer instruction using clickers in a quantum mechanics course. Physical Review Physics
Schuller, M. C., DaRosa, D. A., & Crandall, M. L. (2015). Using just-in-time teaching and peer
doi:10.1097/ACM.0000000000000578
10.1007/s10984-012-9108-4
Sun, J. C., Martinez, B., & Seli, H. (2014). Just-in-time or plenty-of-time teaching? Different
Talley, C. P., & Scherer, S. (2013). The enhanced flipped classroom: increasing academic
10.7709/jnegroeducation.82.3.0339
Wanner, T. (2015). Enhancing student engagement and active learning through just-in-time
teaching and the use of PowerPoint. International Journal of Teaching and Learning in
Appendix
Table 1
Course
Research Sample Locatio Academic
Authors School Course Managemen
Design Size n Domain
t System
Chantoemi
Illinois
et al. Experimental 40
(US)
Undergrad Engineering G-Mail Achievement
(2016)
De et al. Course
Mixed-Methods 235 Malaysia Undergrad Medicine Perceptions
(2014) Website
Liberatore
Colorado Sapling
et al. Experimental 200
(US)
Undergrad Engineering
Learning
Achievement
(2017)
Lopez-
Qualitative 23 Columbia Undergrad Psychology Padlet Engagement
Cupita
(2016)
Illinois
Experimental N/A Undergrad Geography BlackBoard Achievement
Luo (2018) (US)
Common
Marrs & Indiana
Qualitative 750 Undergrad Biology Gateway Perceptions
Novak (US)
Interface
(2014)
Natarajan Kansas
Experimental 626 Undergrad Calculus Javascript Achievement
et al. (US)
(2014)
Osmond et Course
Qualitative 1 US Undergrad Science Perceptions
al. (2012) Website
Riskowski Texas
Experimental 283 Undergrad Physics BlackBoard Achievement
(2015) (US)
Course
Sayer et al. Qualitative 20 US Undergrad Physics
Website
Perceptions
(2016)
Illinois Survey
Schuller et Quantitative 31
(US)
Graduate Medicine
Monkey
Engagement
al. (2015)
Quasi- Polleverywher
Sun et al. Experimental
209 US Undergrad Psychology
e.com
Engagement
(2014)
Talley &
Mixed-Methods N/A US Undergrad Psychology BlackBoard Achievement
Scherer
(2013)
Course
Wanner Qualitative 94 Australia Undergrad N/A
Website
Engagement
(2015)