pubs.acs.

org/jchemeduc Communication

Lessons Learned from the COVID-19 Crisis: Adjusting Assessment

Approaches within Introductory Organic Courses
Andrew P. Dicks,* Barbora Morra, and Kristine B. Quinlan
Cite This: https://dx.doi.org/10.1021/acs.jchemed.0c00529 Read Online

ACCESS Metrics & More Article Recommendations *

sı Supporting Information
See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.

ABSTRACT: This communication describes a variety of virtual student assessment

strategies employed at the University of Toronto during the academic disruption caused by
the 2020 COVID-19 global pandemic. Instructors focused their efforts toward maintaining a
positive learning environment and offering meaningful evaluation methods for students in
Downloaded via CARLETON UNIV on August 4, 2020 at 05:23:52 (UTC).

each of three introductory organic chemistry courses. Assessment schemes were initially
modified in response to moving courses to a virtual platform, and a variety of support
measures were used while students completed the course material and prepared for online
“final assignments”, which in two courses included a virtual rehearsal test. The readiness for
and delivery of online final assignments is outlined (including methods to effectively
maintain academic integrity), and the important roles of graduate student teaching assistants
in successfully completing each course are highlighted. Specific outcomes and reflections are
discussed, including approaches which, with hindsight, were considered unnecessary, and
others that proved to be valuable virtual teaching and assessment tools.
KEYWORDS: First-Year Undergraduate/General, Second-Year Undergraduate, Organic Chemistry, Internet/Web-Based Learning,
Testing/Assessment, Ethics, Mechanisms of Reactions, Synthesis

■ INTRODUCTION
Academic disruptions such as those caused by the 2020
COVID-19 global pandemic have forced instructors to move
courses to virtual platforms, and to adapt their teaching and
assessment strategies in unpredictable and stressful working
environments.1−3 At the University of Toronto, approximately
2000 undergraduates take first-year courses in general and
organic chemistry each year within the Faculty of Arts and
Science (FAS). There are three foundational organic courses in
total: these are CHM 136H (Introductory Organic Chemistry
I), CHM 247H (Introductory Organic Chemistry II), and
CHM 249H (Organic Chemistry). As a first-year level course,
CHM 136H is the prerequisite to the second-year courses and
is aimed at life science students, with a typical enrollment of
800−1000 students per semester. Students may select either of
the second-year level courses, where CHM 247H is also
directed toward life science students (500−600 students per
year) and CHM 249H provides a more rigorous environment
for those interested in pursuing a chemistry degree (60−70
students per year). 4 These three courses all explore
introductory concepts with a focus on (i) the structure,
properties, and reactivity of simple functional groups; (ii) the
mechanisms involved in organic reactions; and (iii) applica-
tions of those reactions in synthesis. As was true at many other
institutions worldwide, each course was significantly impacted
by the COVID-19 pandemic during March and April of 2020.
Appropriate evaluation of student knowledge and under-
standing is a desired teaching outcome, and a responsibility of
© XXXX American Chemical Society and
Division of Chemical Education, Inc.
A J. Chem. Educ. XXXX, XXX, XXX−XXX
chemistry educators.5 During a typical semester, instructors in
CHM 136H, CHM 247H, and CHM 249H primarily assessed
students by designing questions that probed understanding of
course content such as electron flow in reaction mechanisms.
These questions required written, free-response answers that
were marked by hand through a combination of faculty and
graduate student teaching assistants (TAs). However, the rapid
transition of these courses in mid-March 2020 to a 100%
virtual learning environment, coupled with an FAS announce-
ment that all further course assessments had to be conducted
online, meant some different techniques were required.
Necessary adjustments were also recently reported in
administration of the 2020 Advanced Placement (AP)
chemistry examination, and in assessments set by the American
Chemical Society Division of Chemical Education Examina-
tions Institute.6,7 This communication extends previous work
published in areas of evaluation in education (particularly
multiple-choice testing), and associated academic integrity
concerns in the context of short order assessment rede-
sign.5,9−28 The University of Toronto’s Office of Research
Special Issue: Insights Gained While Teaching Chem-
istry in the Time of COVID-19
Received: May 28, 2020
Revised: July 15, 2020
https://dx.doi.org/10.1021/acs.jchemed.0c00529
Journal of Chemical Education pubs.acs.org/jchemeduc Communication

Table 1. Original and Modified Assessment Schemes for CHM 136H, CHM 247H, and CHM 249H
Term Test 1 (%) Term Test 2 (%) Tutorial Quizzes (%) Laboratory (%) Final Examination/ “Final Assignment” (%)a
b
CHM 136H Original 25 or 40 5 20 35 or 50
Modified 25 or 35 N/Ac 10 30d 25 or 35a
CHM 247H Original 15 or 20 15 or 20 N/A 25 40
Modified 25 or 30 25 or 30 N/A 30d 15a
CHM 249H Original 15 15 N/A 35 35
Modified 20 or 25 20 or 25 N/A 35 20a
a
Each original course final examination was replaced by a “final assignment” in the modified assessment scheme. bPossible combination of CHM
136H Term Test 1 and Term Test 2 weightings in original assessment scheme. cCHM 136H Term Test 2 was canceled due to the COVID-19
pandemic. dBased on the best four (of five) completed experiments.

Figure 1. Mechanistic questions included in the CHM 136H online rehearsal assignment.

Ethics has approved the reporting of anonymous student grade encouraging of student learning. For example, TAs would
data in terms of secondary use. provide explanations for challenging concepts in a manner that

■ MODIFIED COURSE ASSESSMENT SCHEMES

The conversion of teaching and evaluation to a purely online
setting (using Canvas as the learning management platform)
required a change in the grading rubrics associated with each of
CHM 136H, CHM 247H, and CHM 249H (Table 1).8 The
original FAS final examination period was replaced by a three-
week final assessment window to allow for a range of
appropriate evaluation tools to be delivered, known as “final
assignments”. The semester marks had two different
weightings within each modified scheme (which required
approval via a majority class vote), and every student was
awarded their highest calculated final course grade.
built on previous knowledge in the course through use of
appropriate figures and schemes. The responses would also
routinely include uplifting remarks (e.g., “great question!”,
“keep up the good work!”, “I hope this helps!”, etc.) that
assisted in creating a supportive learning environment.
Instructor oversight was important to ensure accuracy and
consistency. These methods served to address individual
questions, but also provided a wealth of information for
other students with similar inquiries. In the future, it would be
beneficial to establish separate topic-specific discussion boards
to improve the efficiency of specific content access.
Rehearsal Assignments

■
ONLINE FINAL ASSIGNMENT PREPARATION
Student Support and the Role of Teaching Assistants
Graduate student TAs proved to be a valuable asset by
providing support in preparation for the CHM 136H final
assignment. First, the course instructors created review videos
that were supplemented with short tutorial videos led by
course TAs. These highlighted course content by guiding
students through problem-solving techniques. In addition,
CHM 136H students were encouraged to ask questions via the
Canvas discussion board feature. Questions were answered
daily by both peers and select TAs who contributed thorough
and accurate responses in a manner both supportive and
B https://dx.doi.org/10.1021/acs.jchemed.0c00529
It became logical to redesign the final assignments in the two
large organic chemistry courses (CHM 136H and CHM
247H) to exclusively feature multiple-choice questions. This
created a challenge for instructors to adapt traditional free-
response questions to an online multiple-choice format,
particularly for mechanistic and synthetic problems. Students
were also apprehensive about this new format, since they had
previously been encouraged to complete written problems on
paper throughout each course. To address concerns about the
technical and pedagogical implications of an online multiple-
choice final assignment, some frequently asked questions
(FAQs) and answers were made available to address common
student inquiries (Supporting Information, p S6, S9). In
addition, a not-for-credit rehearsal assignment was offered in
J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
each course, in order to give students an opportunity to
troubleshoot any technical issues and to appreciate how
organic content might be assessed through multiple-choice
testing. The rehearsal assignment for both courses was offered
through the “Quizzes” function of Canvas, using similar
structural and scheduling features as the final assignment
(Supporting Information, p S2). Online practice tests have
previously proved good preparation for traditional pencil-and-
paper assessments in general chemistry classes.9
The format of the rehearsal assignment for CHM 136H and
CHM 247H consisted of several multiple-choice questions,
including one pertaining to academic integrity. Each assign-
ment was available over a wide time frame in order to
accommodate student availability and required completion in
less than 40 min. In CHM 136H, the rehearsal assignment
content was deliberately short and straightforward with a
combination of review material and light-hearted jokes.
Content questions varied in type including “pick the correct
statement” and reaction mechanism problems (Supporting
Information, p S3). The rehearsal assignment mechanistic
questions required students to predict intermediates and draw
electron flow arrows on paper before responding (Figure 1).
This approach closely mirrored what students were expected to
do during an in-person assessment. In total, 665 out of 917
students attempted the rehearsal assignment (consistent with
the number of students who watched the recorded lecture
videos) and completed it in an average of 11.5 min.
Interestingly, the average score was only 53% (typical CHM
136H term test averages are between 60%−75%), and 15% of
students spent less than 5 min before submitting it. This
suggests that a large proportion of students wrote the rehearsal
assignment as intended, whereas some may have used it
exclusively as a technical trial. Overall, the rehearsal assign-
ments in both CHM 136H and CHM 247H were beneficial
exercises that gave instructors and students an opportunity to
familiarize themselves with an online assessment scenario.
Preparing Final Assignment Problems
During the process of writing final assignment questions,
instructors in CHM 136H and CHM 247H enlisted the
assistance of graduate student TAs, who were given sample
multiple-choice questions and topic suggestions. Guidance was
also taken from the pedagogical literature regarding the
definition of learning objectives in helping to generate question
content, stem/response set construction, and other advice.5 In
doing this, emphasis was placed upon design of questions that
avoided assessment of student memorization and recall
(algorithmic/lower-order cognitive skills), and instead focused
on evaluating conceptual/higher-order cognitive skills.10,11 The
TA contributions proved to be extremely valuable in CHM
247H, where a significant proportion of the final assignment
was written by three TAs (Supporting Information, p S10).
However, this mode was less impactful for CHM 136H. A
number of factors likely contributed to this, including a
variability of academic backgrounds and pedagogical experi-
ence among TAs and/or initial guidance provided by
instructors. In addition, a secondary role that both CHM
136H and CHM 247H TAs played was in reviewing the
multiple-choice final assignments to assess level of difficulty,
length, and accuracy, as well as copyediting for consistency and
clarity.
The procedure taken toward designing the final assignment
in CHM 249H (Organic Chemistry) was significantly different
pubs.acs.org/jchemeduc Communication
to that adopted in CHM 136H and CHM 247H for two major
reasons. First, students in this course (65 total) had previously
written two in-person term tests before the transition to virtual
learning/assessment took place. The final assignment was
therefore designed and prepared by the instructor as an online,
noncumulative third term test worth 20% of the course grade
(Table 1) that covered the final 5 weeks of instruction (three
of which were online). Second, the much smaller enrollment in
CHM 249H meant that the final assignment could reasonably
maintain a similar, free-response format as the prior two term
tests, without transitioning to any multiple-choice assessment.
A formal rehearsal assignment was therefore deemed
unnecessary, and instead many practice problems were
provided to students that required structured responses in
preparation for the final assignment.
■ ONLINE FINAL ASSIGNMENT DELIVERY
Structures and Timings
The CHM 136H and CHM 247H online final assignments
were written by students within the April 2020 FAS final
assessment period. Each of them consisted of 30 multiple-
choice questions to be completed within 90 min during a large
time window (CHM 136H: 36 h; CHM 247H: 24 h). This
procedure was recommended to allow for family, scheduling,
and time-zone issues, as well as to allow the respective teaching
teams to resolve any technical concerns. In comparison, the
CHM 249H final assignment was composed of eight free-
response questions to be answered within 3 h during a 24 h
period. Each was administered through the Canvas “Quizzes”
function, with the CHM 249H assignment available as a
downloadable Adobe.pdf file that could be either completed
digitally or printed on paper and answered by hand before
uploading. In retrospect, the lengthy windows granted to
complete the final assignments in all three courses could be
narrowed (e.g., to 12 h). Each long window added unnecessary
stress to the instructional team, and potentially provided more
opportunity for student collaboration. Future online assess-
ments will be offered in discrete, shorter windows, with
alternate sittings offered to students physically located in other
time zones.
One important feature of the CHM 136H and CHM 247H
multiple-choice final assignments was the “scrambling” of
questions, which was straightforward to arrange through
Canvas. This strategy has been reported previously, along
with others such as multiple-choice examination “person-
alization”, and was managed differently in each course.12−15
For CHM 136H, the final assignment was created with 30
question groups and four “versions” of each question as a
method of minimizing potential academic integrity issues. One
question was randomly selected from each group, so the 917
students writing the assignment would largely have unique
versions of it. A similar method was adopted in CHM 247H for
the first five questions where the correct response ((A), (B),
(C), (D), or (E)) was scrambled and following this, the order
of the same 25 remaining questions was randomized. In
contrast to a previous report, no significant impact of response
order was observed on student performance within the first five
questions.16 It was agreed that for future online assessments,
the course instructors would hesitate to write multiple variants
of each question: in general, it has been noted that
“preparation of good quality, challenging online multiple-
choice questions takes time”.17 In addition to the extensive
C https://dx.doi.org/10.1021/acs.jchemed.0c00529
J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education pubs.acs.org/jchemeduc Communication

Figure 2. Sample different CHM 136H online question versions.

effort required to create several versions, the effect of variations

in question groups on student performance was under-
estimated for select questions. For instance, a striking
discrepancy in the CHM 136H final assignment was apparent
for version 1 (item difficulty = 74%, meaning 74% of students
answered correctly) and version 2 (item difficulty = 36%) of a
particular question (Figure 2).5,18 One contributing factor to
this item difficulty difference may be that version 2 was a more
complex multiple choice question, as it contained numerous
options for each answer. These types of questions have been
shown to be less effective as they test analytic skills rather than
the intended learning objective(s).5 Overall, the instructors
acknowledge that extensive effort would be required to
compose question banks with effective questions of equal
difficulty, and that more convenient methods of minimizing
academic integrity issues (such as question and/or answer
randomization) would be more appropriate.
Sample Synthesis Questions
Several multiple-choice questions independently written for
the CHM 136H and CHM 247H final assignments assessed
the student’s ability to recognize and identify functional group
transformations. For example, since alcohol preparation and
reactivity is a topic covered in CHM 136H, a question was Figure 3. CHM 136H online final assignment sample synthesis
constructed regarding nucleophilic addition at a carbonyl question.
group (Figure 3). In CHM 247H, the same type of question
was more challenging as students required thorough under-
Expectations Regarding Academic Integrity
standing of several nucleophilic addition and nucleophilic
substitution reactions, along with related stereochemical A major concern that the course instructors had about moving
considerations (Figure 4). For CHM 249H, the smaller class to online assessment methods was the possibility of academic
size facilitated a final assignment design that was similar to that integrity issues. These have been explored by chemistry
adopted for in-person assessments. Several open-ended, educators in relation to in-person multiple-choice examina-
tions, and the worry of potential “cell-phone based cheating”
advanced tasks were set that were specifically written to
has also been addressed.19−21 Following guidance provided by
prevent students searching their class notes or the course FAS regarding either “open-book” or “closed-book” assess-
textbook for answers, and to instead assess higher-order ments being possible, each course designed an academic
problem-solving capabilities. These involved students propos- honesty honor code regarding what was considered acceptable
ing mechanisms for transformations they had not explicitly behavior in writing the rehearsal assignment (where appro-
encountered before, and designing original synthetic routes priate) and final assignment.22 For example, CHM 247H and
toward target compounds (Figure 5) (Supporting Information, CHM 249H students were informed that they had to
p S14). individually write the final assignment as if they were taking
D https://dx.doi.org/10.1021/acs.jchemed.0c00529
J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Figure 4. CHM 247H online final assignment sample synthesis question.
Figure 5. CHM 249H online final assignment sample synthesis question.
Figure 6. Academic honesty question from the CHM 247H online final assignment.
it under controlled, invigilated conditions in an examination
room. This strategy was adopted to maintain consistency with
the previous two term tests in each course (i.e., no access was
allowed to textbooks, the internet, or other aids) (Supporting
Information, p S9, S13). To reinforce this point, the first
“multiple-choice” question of the CHM 247H rehearsal and
final assignments was a not-for-credit acknowledgment that the
university Code of Behavior on Academic Matters would be
adhered to (Figure 6). In comparison, the CHM 136H final
assignment was designated as being “open-book”, although
collaboration between students was prohibited (Supporting
Information, p S8). Students had only written a single term test
in this course that was held early in the semester, and it was
decided that permitting access to class materials for a “higher-
stakes” final assignment was appropriate (Table 1).
■ ONLINE FINAL ASSIGNMENT OUTCOMES AND
REFLECTION
Given that a combined number of over 1300 students wrote
the three final assignments, there were very few reported
problems. Of 903 students who attempted the CHM 136H
assignment, 18 cases of a technical issue were reported with
pubs.acs.org/jchemeduc Communication
most being resolved within the original 36 h time window. One
issue occurred with the CHM 247H assignment (n = 350) and
three with the CHM 249H assignment (n = 65). The majority
of complications arose through students being unable to load
and view multiple-choice questions within their web browser
window. The implementation of rehearsal assignments was
considered to assist greatly in avoiding technical difficulties
while keeping student anxieties to a minimum.
Although the course instructors and graduate student TAs
had limited experience in designing organic chemistry
multiple-choice assessments, the questions largely met desired
guidelines in terms of item difficulty and discrimination.
Grading automatically took place through Canvas which
additionally produced relevant statistics for each question.
Towns has reported that “a rule of thumb to interpret (item
difficulty) standards is that above 75% is easy, between 25%
and 75% is average, and below 25% is difficult”.5 In addition,
for item discrimination (a measure of how well a question
differentiates between students whose overall assignment
performance demonstrates mastery and those whose perform-
ance does not), a value of above 0.40 is “excellent” (with high
discrimination) and a value between 0.20−0.40 is “good”.5 The
E https://dx.doi.org/10.1021/acs.jchemed.0c00529
J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Figure 7. (a) Item discrimination values; (b) item difficulty standards for the 30 CHM 247H final assignment multiple-choice questions.
Figure 8. CHM 247H Winter 2020 assessment grade distributions (term tests in-person, final assignment online).
item discrimination values and item difficulty standards for the
30 CHM 247H final assignment multiple-choice questions are
shown in Figure 7.
In comparison, the CHM 249H final assignment had to be
graded by hand, but the instructor (A.P.D.) was pleasantly
surprised at how relatively straightforward it was to mark each
assignment electronically, using a digital tablet and stylus. This
technique represented a significant improvement on paper
grading in terms of time spent by eliminating the handling of
physical examination scripts.
Importantly, fears around matters of academic misconduct
were largely unfounded. There are literature examples of
apparent cheating during unproctored virtual examinations,
and a case has been made that environmental factors may
somewhat mitigate deception in online assessments.23−25
However, other reports demonstrate that online quizzes and
examinations do not necessarily increase the cheating instances
found in the student population of particular classes.26−28 Very
little indication of collaboration between students was detected
in any of the three final assignments (e.g., anomalous
performances based on previous course assessments), even
though each assignment was unproctored. Compelling
evidence that dishonesty was not a major issue in CHM
247H is apparent from inspection of the two term test and
online final assignment grade distributions (Figure 8). The
pubs.acs.org/jchemeduc Communication
distribution of grades for the final assignment (average = 60%)
is consistent with that typically observed for a CHM 247H
comprehensive final examination, and does not show a
clustering of higher grades as might be expected with
inappropriate student collusion. One theory to explain this is
that motivation to cheat may have been lower than in previous
semesters, as students were given the option of applying a
“CR” (“credit”) to any undergraduate course impacted by the
COVID-19 crisis during Winter 2020.29 A course with “CR”
attached to it does not have its grade factored into a student’s
cumulative grade point average.
■
CONCLUSION
At the University of Toronto, multiple short order measures
were enacted during the COVID-19 global pandemic within
three introductory organic courses. These were incorporated to
help create positive student learning experiences with realistic
modes of testing. Many lessons were learned, including that
some strategies were unnecessary (e.g., several versions of
multiple-choice questions, lengthy testing windows) while
others worked surprisingly well (e.g., learning management
system technology, rehearsal assignments, and electronic
grading). There was little indication of academic dishonesty
during final assignments, and appropriately functioning
F https://dx.doi.org/10.1021/acs.jchemed.0c00529
J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
multiple-choice items were constructed. Taken together, there
is much to consider as we and others undoubtedly move
toward more virtual teaching, learning, and assessment in the
near future.
■
ASSOCIATED CONTENT
*
sı Supporting Information
The Supporting Information is available at https://pubs.ac-
s.org/doi/10.1021/acs.jchemed.0c00529.
Rehearsal assignment/instructions, final assignment
information for CHM 136H; sample final assignment
questions for CHM 247H and CHM 249H (PDF,
DOCX)
■
AUTHOR INFORMATION
Corresponding Author
Andrew P. Dicks − Department of Chemistry, University of
Toronto, Toronto, Ontario M5S 3H6, Canada; orcid.org/
0000-0001-5456-0212; Email: andrew.dicks@utoronto.ca
Authors
Barbora Morra − Department of Chemistry, University of
Toronto, Toronto, Ontario M5S 3H6, Canada; orcid.org/
0000-0002-0103-2819
Kristine B. Quinlan − Department of Chemistry, University of
Toronto, Toronto, Ontario M5S 3H6, Canada
Complete contact information is available at:
https://pubs.acs.org/10.1021/acs.jchemed.0c00529
Notes
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS
A.P.D. is grateful to graduate students Nicholas Michel, Reggie
Mills, and Bijan Mirabi who contributed significantly towards
the preparation of the CHM 247H final assignment. Mitchell
Winnik, Loise Perruchoud, Scott Browning, and Mark Wilson
are also acknowledged for their assistance and support.
■
REFERENCES
(1) Holme, T. A. Chemistry Education in Times of Disruption and
the Times That Lie Beyond. J. Chem. Educ. 2020, 97 (5), 1219−1220.
(2) Andrews, J. L.; de Los Rios, J. P.; Rayaluru, M.; Lee, S.; Mai, L.;
Schusser, A.; Mak, C. H. Experimenting with At-Home General
Chemistry Laboratories During the COVID-19 Pandemic. J. Chem.
Educ. 2020, 97, 1887.
(3) Potgieter, M.; Pilcher, L. A.; Tekane, R. R.; Louw, I.; Fletcher, L.
Lessons Learnt from Teaching and Learning During Disruptions. In
Research and Practice in Chemistry Education; Schultz, M., Schmid, S.,
Lawrie, G., Eds.; Springer: Singapore, 2019; pp 89−107.
(4) Dicks, A. P.; D’eon, J. C.; Morra, B.; Kutas Chisu, C.; Quinlan,
K. B.; Cannon, A. S. A Systems Thinking Department: Fostering a
Culture of Green Chemistry Practice among Students. J. Chem. Educ.
2019, 96 (12), 2836−2844.
(5) Towns, M. H. Guide to Developing High-Quality, Reliable, and
Valid Multiple-Choice Assessments. J. Chem. Educ. 2014, 91 (9),
1426−1431.
(6) Halford, B. Students will take the Advanced Placement
Chemistry Exam in a Modified Format. https://cen.acs.org/
education/k-12-education/Students-take-Advanced-Placement-
chemistry/98/i18 (accessed 2020−07).
(7) Arnaud, C. ACS Exams Institute Offers ‘Nonsecure’ Versions of
its General and Organic Chemistry Exams. https://cen.acs.org/acs-
pubs.acs.org/jchemeduc Communication
news/ACS-Exams-Institute-offers-nonsecureversions/98/i17 (ac-
cessed 2020−07).
(8) Canvas Learning Management Platform. www.instructure.com/
canvas (accessed 2020−07)].
(9) Prisacari, A. A.; Holme, T. A.; Danielson, J. Comparing Student
Performance Using Computer and Paper-Based Tests: Results from
Two Studies in General Chemistry. J. Chem. Educ. 2017, 94 (12),
1822−1830.
(10) Domyancich, J. M. The Development of Multiple-Choice Items
Consistent with the AP Chemistry Curriculum Framework to More
Accurately Assess Deeper Understanding. J. Chem. Educ. 2014, 91 (9),
1347−1351.
(11) Zoller, U.; Lubezky, A.; Nakhleh, M. B.; Tessier, B.; Dori, Y. J.
Success on Algorithmic and LOCS vs. Conceptual Chemistry Exam
Questions. J. Chem. Educ. 1995, 72 (11), 987−989.
(12) Denyer, G.; Hancock, D. Graded Multiple Choice Questions:
Rewarding Understanding and Preventing Plagiarism. J. Chem. Educ.
2002, 79 (8), 961−964.
(13) Plano, R. J.; Toby, S. Testing, Testing: Good Teaching is
Difficult; So is Meaningful Testing. J. Chem. Educ. 2004, 81 (2), 180−
181.
(14) Carnegie, J. A. Does Correct Answer Distribution Influence
Student Choices When Writing Multiple Choice Examinations? Can.
J. Scholar. Teach. Learn. 2017, 8 (1). DOI: 10.5206/cjsotl-
rcacea.2017.1.11
(15) Manoharan, S. Cheat-Resistant Multiple-Choice Examinations
using Personalization. Comput. Educ. 2019, 130, 139−151.
(16) Tellinghuisen, J.; Sulikowski, M. M. Does the Answer Order
Matter on Multiple-Choice Exams? J. Chem. Educ. 2008, 85 (4), 572−
575.
(17) Ryan, B. J. Line Up, Line Up: Using Technology to Align and
Enhance Peer Learning and Assessment in a Student Centered
Foundation Organic Chemistry Module. Chem. Educ. Res. Pract. 2013,
14, 229−238.
(18) Sevenair, J. P.; Burkett, A. R. Difficulty and Discrimination of
Multiple Choice Questions: A Counterintuitive Result. J. Chem. Educ.
1988, 65 (5), 441−442.
(19) Harpp, D. N. Crime in the Classroom: Analysis Over 26 Years.
J. Chem. Educ. 2018, 95 (2), 338−339.
(20) Harpp, D. N. Crime in the Classroom: Conclusions after 27
Years. J. Chem. Educ. 2018, 95 (10), 1900−1901.
(21) Murphy, K. L.; Holme, T. A. What Might Cell Phone-Based
Cheating on Tests Mean for Chemistry Education? J. Chem. Educ.
2015, 92 (9), 1431−1432.
(22) Burgason, K. A.; Sefiha, O.; Briggs, L. Cheating is in the Eye of
the Beholder: An Evolving Understanding of Academic Misconduct.
Innov. High. Educ. 2019, 44 (3), 203−218.
(23) Harmon, O. R.; Lambrinos, J. Are Online Exams an Invitation
to Cheat? J. Econ. Educ. 2008, 39 (2), 116−125.
(24) Arnold, I. J. M. Cheating at Online Formative Tests: Does it
Pay Off? Internet High Educ. 2016, 29, 98−106.
(25) Fask, A.; Englander, F.; Wang, Z. B. Do Online Exams Facilitate
Cheating? An Experiment Designed to Separate Possible Cheating
from the Effect of the Online Test Taking Environment. J. Acad.
Ethics 2014, 12 (2), 101−112.
(26) Vician, C.; Charlesworth, D. D.; Charlesworth, P. Students’
Perspectives of the Influence of Web-Enhanced Coursework on
Incidences of Cheating. J. Chem. Educ. 2006, 83 (9), 1368−1375.
(27) Hollister, K. K.; Berenson, M. L. Proctored Versus Unproctored
Online Exams: Studying the Impact of Exam Environment on Student
Performance. Decis. Sci. J. Innov. Educ. 2009, 7 (1), 271−294.
(28) Stack, S. The Impact of Exam Environments on Student Test
Scores in Online Courses. J. Crim. Justice Educ. 2015, 26 (3), 273−
282.
(29) University of Toronto Faculty of Arts & Science Credit/No
Credit. www.artsci.utoronto.ca/current/academics/creditno-credit
(accessed 2020−07).
G https://dx.doi.org/10.1021/acs.jchemed.0c00529
J. Chem. Educ. XXXX, XXX, XXX−XXX