Paper Session: CS1 #2
Including Coding Questions in Video Quizzes for a Flipped CS1
Lisa L. Lacher
University of Houston - Clear Lake
Houston, Texas
lacher@uhcl.edu
Yu Zhang
Trinity University
San Antonio, Texas
yzhang@trinity.edu
ABSTRACT
In an effort to improve student performance in a flipped classroom
environment, this paper explores the impact of including auto-
graded coding questions in gate check quizzes associated with
videos for a flipped CS1 course. Previous work showed that having
students complete multiple choice questions that were intended
to verify that they had done the preparation work did not have a
statistically significant impact on outcomes as measured through
written quizzes and exams. In an attempt to engage higher-level
processing of learned information, this work builds on top of that
by adding questions that require students to write short segments
of code for most of the quizzes in addition to doing some multiple
choice questions. We found that students who were given these
coding video quizzes performed better on written assessments,
especially for the final exam.
CCS CONCEPTS
• Social and professional topics → Computer science educa-
tion; CS1;
KEYWORDS
flipped classroom; inverted classroom; flipped learning; CS1; assess-
ments; novice programming; video quizzes; active learning; blended
learning; programming
ACM Reference Format:
Lisa L. Lacher, Albert Jiang, Yu Zhang, and Mark C. Lewis. 2018. Including
Coding Questions in Video Quizzes for a Flipped CS1. In SIGCSE ’18: The
49th ACM Technical Symposium on Computer Science Education, February
21–24, 2018, Baltimore , MD, USA. ACM, New York, NY, USA, 6 pages. https:
//doi.org/10.1145/3159450.3159504
1 INTRODUCTION
The flipped or inverted classroom is considered by some to be one
of the most valuable educational technologies in current use [5].
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
SIGCSE ’18, February 21–24, 2018, Baltimore , MD, USA
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5103-4/18/02. . . $15.00
https://doi.org/10.1145/3159450.3159504
574
SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA
Albert Jiang
Trinity University
San Antonio, Texas
xjiang@trinity.edu
Mark C. Lewis
Trinity University
San Antonio, Texas
mlewis@trinity.edu
The basic idea behind flipped learning is that it reverses the learning
process from the traditional classroom. In a flipped class, students
review new content, often by watching videos which contain nor-
mal lecture material, prior to class, and then use class time to work
on problem solving exercises, projects, and homework assignments
in order to promote higher-order learning. Continuing advances in
technology make it easier than ever to create and post content for
students to view and interact with on many platforms and in many
locations, which led to an increase in the use of flipped classrooms.
New learning communities, such as SOPHIA and the Flipped Learn-
ing Network, have been established to help interested educators
implement the flipped model by providing them with knowledge
and resources. One indication of the growing interest in flipped
learning can be seen by membership surveys in these communities.
A survey conducted by both communities indicated a ten-fold in-
crease in membership. It also found that the number of educators
that flipped at least one lesson has almost doubled between 2012
and 2014, and that 96% of educators who have flipped a lesson
would recommend the method to others [18].
The flipped classroom has certainly been gaining momentum
within higher education; especially within Computer Science, as
it allows students to spend more of the valuable class time doing
hands-on activities instead of passively listening to a lecture[7].
This increased adoption is not surprising as research suggests that
millennial students prefer the interactive learning that the flipped
model provides [17]. Student engagement is just one of the benefits
touted by many flipped classroom researchers[15]. Many studies
show other promising benefits of the flipped model beyond student
engagement. Some of these benefits include students in the flipped
classrooms outperforming students in lecture based classrooms[2,
8], more positive attitudes, better learning habits, development of
critical thinking skills [9], and the ability to cover more course
material [15].
However, there are still many challenges with the flipped model.
Not all studies actually show improved student performance [3].
Another challenge is that the flipped model requires students to
be more responsible for their own learning and not all students
are comfortable with or prepared for that responsibility [20]. Uni-
versity freshman, in particular, need much more motivation from
instructors to take responsibility [7]. Yet other issues are a lack of
student accountability to complete the instruction that happens
outside of class and the inability to monitor how well the students
have understood the pre-class instructional materials [6]. Thus, we
Paper Session: CS1 #2
can see that the flipped classroom method provides much promise,
but there are still many pitfalls that mainly center around student
preparedness. The effectiveness of the flipped method depends on
how well prepared the students show up to class. If a sufficient
number of students do not adequately prepare, then the instruc-
tor has few other options than to revert back to the traditional
lecture format. For this reason, instructors who use the flipped
method would benefit by having a mechanism that increases the
likelihood that students will show up to class having completed the
recommended preparation work. It is necessary that they will have
gained enough prior knowledge so that they will have more time
to conduct higher level cognitive activities during class time.
This work builds on [10, 12], which explored the value of simple
gate-check quizzes using multiple choice questions in a CS1 flipped
classroom to help the students be more accountable and prepared
for class. Both studies found that the multiple choice question
style of gate-check quizzes was not effective in helping students
adequately prepare. In this paper, we discuss a study that used
coding questions, in addition to multiple choice questions, for the
gate-check quizzes. Our goal was to see if questions that required
the student to invoke higher forms of thinking and apply knowledge
that they should have gained by watching videos would be more
beneficial than simply memorizing or looking up answers.
2 BACKGROUND
Bloom’s taxonomy differentiates between cognitive skill levels. It
calls attention to learning objectives that require higher levels of
cognitive skills that lead to deeper learning, and transfer of both
knowledge and skills to a greater variety of tasks and contexts
[1]. The fundamental idea of the flipped learning model is to have
students do the lower levels of cognitive work, such as gaining
knowledge by remembering and understanding, outside of class.
Then instructors can attempt to more positively impact the students’
higher level cognitive work during class time by using more active
learning activities where students apply, analyze and create, whilst
getting immediate assistance and feedback from the instructor or
their peers. However, in order to achieve the potential of the higher
cognitive levels students must take the initiative outside of class
to prepare and achieve a sufficient level of understanding of the
material. Because most CS1 courses are composed of freshmen, and
they often have lower levels of learning readiness [7, 20], educators
must use a mechanism to assist in their motivation.
Some researchers have used gate-check quizzes in flipped class-
rooms to motivate students to prepare [6], but little is known about
the actual makeup of the quizzes. A gate-check quiz is administered
after the student has consumed the materials to give the student
feedback about their level of understanding before they would enter
the classroom – to let them know if they are ready to go through
the gate. Thus the feedback from this quiz should be immediate.
There is a lack of research about the type of questions that could
be used in gate-check quizzes to improved learning outcomes in a
flipped classroom. Zhang et al.[21] suggest that interactive quizzes
(multiple-choice style) within videos can be used to improve learn-
ing effectiveness. However, Horton [8] found that "Students who
passively watch the videos and click through the in-video quizzes
without much thought may get a false sense of mastery over the
575
SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA
material". Also, not all instructors have the technology available
to create interactive video quizzes or insert them into preexisting
videos. Stanger-Hall [19] found that using constructive response
questions in addition to multiple choice questions significantly im-
proved students’ performance. Constructive response requires more
thought and according to [1], students who engage in tasks that
require higher levels of cognitive skills engage in deeper learning
which aids in their understanding of the material.
We preformed a previous study about the effectiveness of gate-
check quizzes in a CS1 class that used online multiple choice ques-
tions. Although it was found that the multiple choice question style
of gate-check quiz was not effective in helping students adequately
prepare to learn (measured by grade performance on written quizzes
and exams) in a flipped classroom; there was some evidence that it
might have helped raise the grades of students who have a propen-
sity toward surface learning [10, 12].
We believe that the limited nature of multiple choice quiz ques-
tions did not adequately test the students’ understanding and pro-
vide the necessary feedback to them so that they knew what they
did and did not understand. It also did not stimulate the students’
motivation to learn and understand the course material. In other
words, multiple choice questions did not promote deep learning.
The multiple choice quiz style was a surface learning tool, not
something that prompted students to think more deeply, make
connections, or question their assumptions. Based upon what we
learned in that study, we conceived the idea of adding questions
with more cognitive complexity. We felt that including coding ques-
tions, which necessitate some application or simple analysis of ideas
or concepts introduced during the videos, would engage a higher
level of cognitive thinking that would produce deeper learning and
understanding. Because writing program code is one of the primary
ways CS1 students communicate their understanding of material,
we wanted questions that required students to write programming
code solutions to simple problems. Students need to apply some
programming concepts in order for students to fully understand
these concepts [4].
3 STUDY DESIGN
This study looks at five sections of CS1 taught by two instructors in
the fall semester of 2015. All sections used a flipped approach where
students watched videos outside of class, and then spent class time
applying the knowledge they should have acquired from the videos.
We designed the study to answer the question of whether the use of
coding gate-check quizzes was helpful in improving the students’
learning as measured by the in-class quiz and exam scores. We
administered pre-tests in order to assess each student’s computer
programming aptitude. The study compares the performance of stu-
dents that were required to complete pre-class coding quizzes that
provided immediate feedback to students that were not required to
take such quizzes.
3.1 Hypotheses
For this study we had two hypotheses related to requiring the cod-
ing style gate-check video quizzes as a means of assisting students
learning preparation before coming to class.
Paper Session: CS1 #2
• Hypothesis 1: Students taking the coding gate-check video
quizzes have higher grades on the assessments than students
who do not have to take the coding quizzes.
• Hypothesis 2: Students with lower programming aptitudes
benefit more from coding gate-check quizzes than students
who have medium or high programming aptitudes.
The rationale behind the first hypothesis is that making the stu-
dents apply knowledge learned from the videos would stimulate
deeper thinking to help them really understand programming con-
cepts. Successfully completing a coding question should help better
prepare the student to engage in the active learning activities that
occur in the classroom so that they get more out of that learning
experience and earn better grades. As coding ability is important
to learning the fundamental concepts in CS1, this type of question
can better inform the instructor about the students’ abilities. This
type of knowledge can help the instructor design a better learn-
ing experience for the classroom to assist in student learning. You
cannot get that type of feedback from multiple choice questions.
We based the second hypothesis on the belief that students with
high programming aptitudes would generally do well regardless of
pre-class preparation and, as such, would not benefit much from
the coding quizzes. On the other hand, forcing the lower aptitude
students to work harder at their preparation by writing small pieces
of code that force them to focus on details of both syntax and
semantics would help them be more successful in the class.
3.2 Participating Subjects
There were eighty-five undergraduate students enrolled in five sec-
tions of CS1 at Trinity University. Trinity University is a small,
selective liberal arts college with 2500 undergraduate students,
during the fall semester of 2015. The courses used [14] and the as-
sociated videos that are available on the author’s YouTube channel
[13]. Prior to the beginning of classes, the instructors randomly
choose which class sections would use video quizzes and which
class sections would not use video quizzes. Sixty-nine of the stu-
dents participated in the study and completed quizzes and tests:
forty-one participants took the video quizzes and twenty-eight did
not take video quizzes. Instructors urged every section of students
to watch the videos and/or read the textbook before attending class.
Instructor A taught two video quiz sections and one non-video
quiz section. Instructor B taught one video quiz section and one
non-video quiz section. We would have liked to do a statistical
analysis of the impact for each instructor separately, but the sample
size would have been too small.
3.3 Artifacts
The classes used identical in-class quizzes for the quizzes that were
included in the study. There were minor differences in the material
presented late in the semester between the instructors. As a result,
Instructor A offered a sixth quiz, which was not included in the
study. Each student took a midterm and a final at the same point
in the semester. Students had advanced knowledge of the dates of
all quizzes and exams. Two of the sections taught by Instructor B,
one with the video quizzes and one without, had three 50 minute
periods each week. The other sections, taught by Instructor A, each
had two 75 minute periods each week. To compensate for this, the
576
SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA
midterm tests were identical for the first seven questions and the 75
minute sections had an extra three questions. Although the majority
of the final test questions were identical between all sections, there
was one question that was different due to the pace of the classes.
Instructor B’s classes ended up one week behind Instructor A’s
classes. Again, there was always a balance between courses with
the video quizzes and courses without the video quizzes using
different versions. The fact that both courses with and without the
video quizzes had identical exams balances out the overall statistics
for this work.
We created one rubric for each quiz to aid with consistency in
grading. One teaching assistant, using the rubric as a guide, graded
all the quizzes. Instructors graded the tests, with each instructor
being responsible for a certain set of questions. This ensured con-
sistency in grading across all tests. Each day in class, the students
worked on writing code solutions for a variety of problems posed
by the instructor. There was consistency in the problems posed
between sections.
The students also completed four homework assignments. How-
ever, those homework assignments were not included in the study
because students who put forth time and effort basically received
full or nearly full credit. Because we did not track time and ef-
fort spent on homework assignments, those assignment grades do
not provide a useful way to analyze the impact that the coding
gate-check quizzes would have had on them.
3.4 Experimental Procedure
To evaluate the hypotheses posed, the study contained a control
group and an experiment group. We give the details of the experi-
ment steps below.
3.4.1 Course Assessments. Three of the sections had ten percent
of the grade determined by short quizzes that students completed
after watching the videos and before their scheduled class meeting
times. We give details of these quizzes, including the nature of the
coding questions, below. It was felt that ten percent of a grade was
sufficient incentive to ensure that students took the gate-check
quizzes seriously. To determine a composite grade for this study, in-
class quiz scores contributed twenty-five percent, and the midterm
and final contributed equally to the remaining seventy-five percent
of the grade. Actual course grades used a different formula to in-
clude assignments and other coding activities not directly factored
into this work.
3.4.2 Pre-Test. Students were asked at the beginning of the se-
mester to complete a computer programming aptitude test created
by the University of Kent [16]. It consists of twenty-six questions
composed of numerical problem solving, logical reasoning, atten-
tion to detail, pattern recognition, and the ability to follow complex
procedures. As it does not require students to have any program-
ming knowledge, it is appropriate for CS1 students. Numerical
problem solving is similar to the logical thinking and trouble shoot-
ing skills required in programming. Pattern recognition is necessary
in understanding the representations of symbols and procedures.
The ability to follow complex procedures is necessary to trace code.
Students had forty minutes to complete this test. The University of
Kent breaks scores into three groups. Scores >= 19 are considered
Paper Session: CS1 #2
Table 1: Grade Results: Quiz n=41, NoQuiz n=28.
Q1 Q2 Q3 Q4
QuizAvg 8.09 5.94 3.56 6.91
QuizMedian 8.5 6 3 7 6
NoQuizAvg 7.17 5.07 2.89 5.37
NoquizMedian 7 5 2 4.5
p 0.075 0.142 0.364 0.039
high-aptitude; scores 13-18 are considered average aptitude, and
scores 12 or below are considered low aptitude.
3.4.3 Gate-check Quizzes. The “video quizzes” included ques-
tions that required the students to write simple expressions or small
functions as well as multiple choice questions when suitable coding
questions could not be written for a topic. The sections took be-
tween twenty and twenty-two of these video quizzes. The quizzes
were generally available at least two days before they were due.1
For the twenty-two video quizzes there were eighteen coding
questions and thirty-three multiple choice questions. The majority
of the coding questions were before the midterm when students
were learning basic coding concepts like conditionals, functions,
and loops. There were more multiple choice questions in the second
half of the semester for topics like GUIs, graphics, and sorting.
Students could only answer each multiple choice question once,
but the coding questions allowed for unlimited submissions. There
were more multiple choice questions than coding questions because
if a topic lent itself to coding questions, there would generally only
be one or possibly two coding questions on the quiz, while a quiz
with only multiple choice questions would have three or four such
questions for the same amount of material.
In addition to allowing unlimited submissions for the coding
questions, students could ask for assistance from the professor2
or work with other students on the coding question. Allowing un-
limited attempts and using the professor and/or other students as
resources helps minimize anxiety. Practicing programming involves
trial and error and allowing multiple submissions permits trial and
error. Allowing students to talk to the professors provides students
another source of feedback to help with their learning and this is
particularly advantageous if the students do not understand what is
wrong with their submissions. The motivation for allowing commu-
nications with other students is to potentially enable collaborative
learning, but our experience is that students do not often collabo-
rate on the quizzes. Once again, the purpose of the pre-class video
quiz questions is to make sure the students put in the necessary
preparation so that they can get the most out of the active learning
exercises they will be doing in class.
Here are four sample coding questions used in the video quizzes
in the first half of the semester.
• In math you would write 0<x<5 to indicate that x is between
0 and 5. What would you write in Scala to express this same
1 They were available for a longer period of time if there was a weekend between
classes.
2 Professors in the Trinity CS department have open-door office hours, and are highly
available through email, so all students had the ability to ask questions, regardless of
whether or not they were in the video quiz group.
577
SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA
Q5 Midterm Final Combined Grade
6.03 49.54 59.39 66.4
65 75 68.5
5.32 48.0 49.22 57.0
5.25 63 60 53
0.427 0.350 0.030 0.00072
restriction? Write the proper expression and assume x is a
Double.
• Write an expression that assumes there is a List[Int] called
nums which calculates the sum of the even values in the List.
• Write a function called sumOdds that takes n:Int and re-
turns the sum of all the odd numbers between 1 and n inclu-
sive. So if you call sumOdds(8) it should return 16 because
1+3+5+7=16.
• Write a function that takes a List[Int] and returns the index
of the first number divisible by 5. If there is no such number,
return -1.
The coding questions were auto-graded by executing student code
on a suite of randomized inputs and checking their results against a
correct solution. If students submitted incorrect answers for a cod-
ing question, they were only told that their response was incorrect,
with no details on the nature of the error. Students were strongly
encouraged to test their code on their own before submitting. Mul-
tiple submissions were common. A section with sixteen students
generally had 50+ submissions and some problems had closer to
100 attempts for a given problem. Once a student got the problem
correct, going back to the quiz showed them the code from their
correct submission as un-editable text.
3.4.4 In-class Quizzes and Tests. The questions on the in-class
quizzes and tests used for assessment were very different from the
questions on the video quizzes. There were three types of questions:
short answer, coding, and tracing. We used an equal amount of
each type. There were no multiple choice questions, and the coding
questions were typically larger in scope than the coding questions
on the video quizzes. The medium used for the in-class quizzes and
tests was paper.
4 ANALYSIS AND RESULTS
We carried out statistical analyses on the data collected during
the study and tested the described hypotheses to determine the
usefulness of the coding gate-check quizzes in improving student
understanding as proven out by grades. Independent samples t-tests
were performed to determine if differences were statistically signifi-
cant. We selected an alpha value of 0.05 for judging the significance
of the results.
4.1 Hypothesis 1
The students taking the coding gate-check video quizzes have higher
grades on the assessments than students who do not have to take the
coding quizzes.
Table 1 shows the numeric values for means and medians of all
of the grades collected for both the video quiz and non-video quiz
Paper Session: CS1 #2 SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA

sections. It also shows the p value for a t-test to see if the differences
in the means are statistically significant. While the difference for 90
individual grades are only significant for quiz four and the final
exam, the group that took the video quizzes had a higher mean 80

and median for all of the grades, so the combined grade shows a 70
advantage for the group that was taking the video quizzes.
60
Note that this is in direct contrast to the results of [10, 12] where
the gate-check video quizzes only included multiple choice ques- 50 Instructor A - vid
Instructor B - vid
tions. The results here support the hypothesis, indicating that the
40 Instructor A - no vid
addition of coding questions to the video quizzes was the primary Instructor B - no vid

factor in improving the effectiveness of the video quizzes as mea- 30

sured by student grades. 20

The final showed a much larger difference than the midterm. We
believe that this is not just an anomaly due to the small number 10

of students, but a real effect that indicates that the true benefit 0
Quiz #1 Quiz #2 Quiz #3 Test #1 Quiz #4 Quiz #5 Test #2
of adding coding questions to the video quizzes was fully mani-
fest in the second half of the semester, even though many of the
video quizzes in the second half of the semester often lacked coding
questions. This is because the material in CS1 is very implicitly
cumulative. Each topic has a tendency to build on the ones before Figure 1: This figure shows the median grades for each quiz
it. The CS1 curricula used in this study also tends to get more chal- and test broken out by instructor and whether or not the sec-
lenging for many students after the midterm. Students who have an tion used the video quizzes. The sample sizes for each group
inferior mastery of the material from the first half of the semester are as follows: Instructor A vid - 28, Instructor A no-vid - 14,
get lost and struggle more in the second half of the semester. So Instructor B vid - 13, Instructor B no-vid - 14.
while students who had not fully mastered the material might have
been able to sufficiently study enough to get by the midterm, the 100
90
80
second half of the semester made their lack of mastery more obvious
70
which, most likely, caused them to fall further behind in the second
60
Course Grade

half of the semester. We believe that the coding questions on the

Quiz
video quizzes helped students achieve mastery, which paid off by 50
No Quiz
the end of the course. It definitely presented a good opportunity to 40
30 Linear (Quiz)
make students code more which should have had a positive impact
20 Linear (No Quiz)
10
on their learning.
0
Given the small sample size, we cannot do statistical analysis
on groups broken out by instructor, but there were no significant 0 10 20 30
effects due to a difference in instructors. To illustrate this, Figure Aptitude

1 shows the mean grades for each of the written evaluations in

the order students completed them during the semester, broken
out by groups for instructor and whether or not they took the
coding video quizzes. In this plot, there are 14 pairs of vid/no-vid
groupings, seven for each instructor. It is worth noting that of those
14 pairs, there are only three cases where students who did not
take the video quizzes scored higher than students who did take the
quizzes for the same instructor. Those cases are quizzes #2 and #3
for Instructor B and quiz #4 for Instructor A. In all three cases, the
margin was small. This plot shows that while there were differences
between the instructors on the various written assessments, the
overall impact of the coding video quizzes is not generally instructor
dependent.
4.2 Hypothesis 2
Students with lower programming aptitudes benefit more from coding
gate-check quizzes than students who have medium or high program-
ming aptitudes.
Figure 2 shows the course grades of students in the two groups
plotted against aptitude along with trend lines. Our hypothesis was
Figure 2: Cumulative student grades plotted against apti-
tudes. The fit for the video quiz group is y = 2.7027x + 20.449
with R 2 = 0.39. The fit for non-video quiz group is y =
1.9053x + 27.94 with R 2 = 0.1724. In both groups there is a
positive correlation between aptitude and grades.
that being forced to do regular coding on the video quizzes would
provide the most benefit to the low aptitude students. Though it is
hard to draw strong conclusions here given the general dearth of
low aptitude students, the trend lines indicate that this hypothesis
was incorrect. Instead, it appears that the low aptitude students
performed at roughly the same level between the two groups, and
it was the higher aptitude students who received the most benefit.
Figure 2 is significant for other reasons as well. First, both groups
show a clear positive slope.3 This would be expected for a valid
aptitude test as high aptitude students should be more likely to do
3 The slope of the non-video quiz group is effectively determined by the one low
aptitude outlier. Removing that point causes the trend line to have a slightly negative
slope over the small range of aptitudes in the sample.

578
Paper Session: CS1 #2
well on standard assessments. This result basically shows further
support for the University of Kent aptitude test being a valid met-
ric of student aptitude that is independent of any programming
background.
In addition, given the smaller number of students involved in
this study, it would have been possible for students to be unevenly
distributed between the two groups based on aptitude. That would
have been a challenge to the validity of all the conclusions presented
here. This plot shows that the aptitude distribution between the
two groups was roughly equivalent. If anything, the quiz group
had more students with lower aptitude, so the fact that they had
higher overall scores appears to indicate that the use of the coding
video quizzes provided a benefit, even if it does not appear to have
benefited the low aptitude students as we had predicted.
5 CONCLUSIONS
This work found that student grade outcomes benefited in a flipped
classroom when using gate-check quizzes that force the students to
write code related to the topic of the new material. This is in contrast
to previous work where the gate-check video quizzes aimed at
simply verifying that students watched the videos, not whether they
had learned enough from them to apply that knowledge, included
only multiple choice questions.
This benefit was not uniform for all students. Students with
higher aptitudes for CS seemed to accrue most of the benefits,
contrary to our hopes and expectations. Still, we feel that these
results provide a strong motivation for including these techniques
in any type of inverted classroom because we did see positive results.
Unfortunately, it does not seem to provide significant benefits to
the population of students who probably need it the most. More
work needs to be done to see how lower aptitued students can
be helped. Analysis of this study data related to how to deal with
low aptitude students in introductory computer science courses is
discussed presented in [11].
There were several limitations with this study. The study was
performed at a small, private liberal arts university and the class
sizes are constrained to rooms that seat 21. We call on others with
larger student populations to continue this work. We also realize
that we required the experimental group to take video quizzes and
that there was no equivalent effort or work required for the control
group. We took this approach because we wanted to create circum-
stances as similar as possible to previous work so that we could
more accurately measure this new gate-check quiz approach. This
is why we brought in comparisons from previous work. Previous
work found the same confounding variable - the study required
the experimental group to take video quizzes and there was no
equivalent work required for the control group. Yet, the evidence
suggests that the addition of coding questions to the gate-check
quizzes had a more positive impact on students than just using
579
SIGCSE’18, February 21-24, 2018, Baltimore, MD, USA
multiple choice questions. Another threat includes the fact that we
had two different instructors and instructors may have different
teaching styles. However, to achieve as much balance as possible,
each instructor had students who took the video quizzes and stu-
dents who did not take the video quizzes. Plus, all instructors used
the flipped classroom approach and all student watch the same
video lectures.
REFERENCES
[1] N. E. Adams. Blooms taxonomy of learning objectives. Journal of the Medical
Library Association, pages 152–153, Jul 2015.
[2] A. Amresh, A. R. Carberry, and J. Femiani. Evaluating the effectiveness of flipped
classrooms for teaching cs1. In 2013 IEEE Frontiers in Education Conference (FIE),
pages 733–735, Oct 2013.
[3] W. G. Bowen, M. M. Chingos, K. A. Lack, and T. I. Nygren. Interactive learn-
ing online at public universities: evidence from a six-campus randomized trial.
Journal of Policy Analysis and Management, 33(1):94–111, 2014.
[4] S. Cummins, A. R. Beresford, and A. Rice. Investigating engagement with in-
video quiz questions in a programming course. IEEE Transactions on Learning
Technologies, 9(1):57–66, 2016.
[5] T. Economides. The state of the art in educational technology. In Global Human-
itarian Technology Conference (GHTC), 2013 IEEE, pages 285–287, Oct 2013.
[6] J. Enfield. Looking at the impact of the flipped classroom model of instruction
on undergraduate multimedia students at csun. TechTrends, 57(6):14–27, 2013.
[7] Y. Hao. Exploring undergraduates’ perspectives and flipped learning readiness
in their flipped classrooms. Computers in Human Behavior, 59:82–92, 2016.
[8] D. Horton and M. Craig. Drop, fail, pass, continue: Persistence in cs1 and beyond
in traditional and inverted delivery. In Proceedings of the 46th ACM Technical
Symposium on Computer Science Education, SIGCSE ’15, pages 235–240, New
York, NY, USA, 2015. ACM.
[9] S. C. Kong. Developing information literacy and critical thinking skills through
domain knowledge learning in digital classrooms: An experience of practicing
flipped classroom strategy. Computers & Education, 78:160–173, 2014.
[10] L. Lacher and M. C. Lewis. The value of video quizzes in a computer science
flipped classroom: An empirical study. In The 2014 International Conference on
Frontiers in Education: Computer Science & Computer Engineering, pages 94–100.
CSREA Press, 2014.
[11] L. L. Lacher, A. Jiang, Y. Zhang, and M. C. Lewis. Aptitude and previous experience
in cs1 classes. In Int’l Conf. Frontiers in Education: CS and CE | FECS’17, pages
87–95, 2017.
[12] L. L. Lacher and M. C. Lewis. The effectiveness of video quizzes in a flipped
class. In Proceedings of the 46th ACM Technical Symposium on Computer Science
Education, pages 224–228. ACM, 2015.
[13] M. C. Lewis. Instructional youtube videos. https://www.youtube.com/channel/
UCEvjiWkK2BoIH819T-buioQ.
[14] M. C. Lewis. Introduction to the art of programming using scala, 2012.
[15] G. S. Mason, T. R. Shuman, and K. E. Cook. Comparing the effectiveness of an
inverted classroom to a traditional classroom in an upper-division engineering
course. IEEE Transactions on Education, 56(4):430–435, 2013.
[16] U. of Kent. University of kent. https://www.kent.ac.uk/careers/tests/
computer-test.htm.
[17] C. R. Phillips and J. E. Trainor. Millennial students and the flipped classroom.
Journal of Business and Educational Leadership, 5(1):102, 2014.
[18] Sophia.org. Growth in flipped learning. https://www.sophia.org/
flipped-classroom-survey, 2014. [Online; accessed 22-August-2016].
[19] K. F. Stanger-Hall. Multiple-choice exams: An obstacle for higher-level thinking
in introductory science classes. In CBE life sciences education, 2012.
[20] J. F. Strayer. How learning in an inverted classroom influences cooperation,
innovation and task orientation. Learning Environments Research, 15(2):171–193,
2012.
[21] D. Zhang, L. Zhou, R. Briggs, and J. Nunamaker. Instructional video in e-learning:
Assessing the impact of interactive video on learning effectiveness. Information
and Management, 43(1):15–27, 1 2006.