See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/284786816

MCQ tests in Advanced Trauma Life Support (ATLS ©): Development and
revision

Article  in  Injury · November 2015

DOI: 10.1016/j.injury.2015.11.024

CITATIONS READS

6 26,473

6 authors, including:

Jacqueline Bustraan John Kortbeek

Leiden University Medical Centre The University of Calgary
12 PUBLICATIONS   130 CITATIONS    118 PUBLICATIONS   3,939 CITATIONS   

SEE PROFILE SEE PROFILE

Inger B Schipper
Leiden University Medical Centre
233 PUBLICATIONS   5,535 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Warrior trial View project

MTV crashes View project

All content following this page was uploaded by Jacqueline Bustraan on 19 August 2020.

The user has requested enhancement of the downloaded file.

G Model
JINJ-6511; No. of Pages 4

Injury, Int. J. Care Injured xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Injury
journal homepage: www.elsevier.com/locate/injury

MCQ tests in Advanced Trauma Life Support (ATLSß): Development

and revision
Jacqueline Bustraan a,*, Walter Henny b, John B. Kortbeek c,1, Karen J. Brasel d,
Marzellus Hofmann e, Inger B. Schipper f
a
PLATO, Leiden University, P.O. Box 9555, 2300 RB Leiden, The Netherlands
b
ATLS Netherlands, Tilburg, The Netherlands
c
Department of Surgery, University of Calgary and Alberta Health Services – Calgary Foothills Medical Center, Department of Surgery, 10th Floor N. Tower,
1403-29th Street, NW Calgary, AB, Canada T2N 2T9
d
Department of Surgery, Medical College of Wisconsin, Milwaukee, USA
e
Faculty of Health Witten-Herdecke University, Alfred-Herrhausen-Str 50, 584 Witten, NRW, Germany
f
Department of Trauma Surgery, Leiden University Medical Center, Leiden, The Netherlands

A R T I C L E I N F O A B S T R A C T

Article history: Background: In Advanced Trauma Life Support (ATLSß) courses, multiple choice question (MCQ) tests are
Accepted 15 November 2015 used to assess student’s post course knowledge. As part of the ninth Edition Revision Process, existing
MCQ tests were reviewed and revised by an International MCQ Revision group. The aim of this study was
Keywords: to evaluate the revision procedure and its effects.
ATLS Methods: Based on psychometric data and evidence based guidelines for adequate MCQ item and test
Assessment development, a detailed stepwise approach was determined and followed to evaluate the existing MCQs,
Trauma education
and to guide test item revision or replacement.
MCQ
Results: The MCQ Revision group composed three new draft test versions comprising of 40 MCQs each.
These were beta-tested among ATLS Instructors in various countries involved in ATLS. Psychometric
analysis demonstrated that a minority of MCQ items required revision to create three equally balanced
tests. After these final adjustments, a new set of three validated MCQ tests was available for use in 9th
edition ATLS provider courses. Beta testing was performed using instructors but not students. The failure
rate amongst students of ATLS provider courses increased significantly after introduction of the new
MCQ tests.
Conclusion: ATLS tests were revised and updated using current evidence based guidelines and
psychometric analysis. Difficulty of the tests was not initially beta-tested on students. Increasing test
item discrimination and quality resulted in lower test scores by students.
ß 2015 Elsevier Ltd. All rights reserved.

Background course, summative assessment is performed and is based on MCQ

Currently, the Advanced Trauma Life Support (ATLSß) course
for doctors is taught in more than 60 countries worldwide. It is a
widely accepted standard for the initial care of trauma patients
[1]. During ATLS courses, the students undergo serial formative
assessment during varied skills stations. At the completion of the
* Corresponding author at: PLATO, Leiden University, P.O. Box 9555, 2300 RB
Leiden, The Netherlands. Tel.: +31 71 5268364.
E-mail addresses: j.bustraan@lumc.nl (J. Bustraan), whenny@online.nl
(W. Henny), kbrasel@mcw.edu (K.J. Brasel), Marzellus.Hofmann@uni-wh.de
(M. Hofmann), b.schipper@lumc.nl">i.?b.schipper@lumc.nl (I.B. Schipper).
1
Tel.: +1 403 944 1697.
testing as well as on performance tests during simulated patient
assessments. As knowledge is important in medical competence
and a necessary prerequisite for successful problem solving, the
main goal of the post course MCQ test is to assess the student’s final
level of knowledge. It is assumed that students will perform better
on this test, as compared to the pre-course test, once they had the
opportunity to elaborate on the course content in interactive
lectures and were challenged to apply their knowledge in ATLS
skills stations and Initial Assessments, using the systematic
approach of patients [2].
Each ATLS MCQ test consists of 40 single best-option questions,
which can be regarded as the most versatile and widely used
multiple-choice question type [3]. For the final testing ATLS

http://dx.doi.org/10.1016/j.injury.2015.11.024
0020–1383/ß 2015 Elsevier Ltd. All rights reserved.

Please cite this article in press as: Bustraan J, et al. MCQ tests in Advanced Trauma Life Support (ATLSß): Development and revision.
Injury (2015), http://dx.doi.org/10.1016/j.injury.2015.11.024
G Model
JINJ-6511; No. of Pages 4

2 J. Bustraan et al. / Injury, Int. J. Care Injured xxx (2015) xxx–xxx

employs three different test versions in order to be able to vary the feedback before officially launching the new product. Compose
tests between consecutive courses and course-sites. In general, one three tests of 40 questions each. Beta-test again.
of the three tests is chosen to serve as initial final test. One of the 6. Perform final check on match between the content as presented
other two is used as retest for students who failed the first time and in the final version of the 9th edition manual and the final MCQs.
occasionally, the third is used as second retest. Since the tests are Make last adjustments if necessary before launching the tests
not publicly available, they can be used repeatedly. worldwide.
As with any assessment instrument, MCQ tests do have specific 7. Collect international data on test results during 1 year.
limitations and disadvantages. Yet, if well-constructed and peer
reviewed, MCQ tests can meet important educational standards. All International MCQ Revision group members were provided
Good quality MCQs can test more than the mere recall of with literature [8,9] on MCQ construction. They received instruc-
knowledge. Benefits of automated marking and a potentially high tions and feedback at step 1–3 by the Chair and Educator in the
reliability at low costs make MCQs a viable option [4]. Although group.
guidelines for developing adequate MCQ items are available, the
construction of good MCQ tests remains a challenge.
Results
A pilot study in 2009, analysing data from 190 Dutch ATLS
Instructors updating for the 8th edition, demonstrated that the
Following the MCQ development schedule, the results of the
tests used until then were not optimally balanced within and
review and development process are presented stepwise.
between tests. To determine whether the results were applicable
and relevant for ATLS students, a second identical psychometric
1. The test matrix for the 9th edition MCQ’s had the following
study was performed in 2010, using student data from
characteristics, some of which were based on assumptions:
Switzerland, Israel and the Netherlands. The data from this study
 The ATLS manual covers all important topics.
were compared to the data from the extensive students’ database
 All 13 chapters/topics are equally important.
of the American College of Surgeons (ACS). The specific methods
 Students have to show their mastery of all topics on two
and outcomes of the psychometric analysis of these studies were
levels: knowledge and application of knowledge using an
presented at several ATLS International Conferences between
ABCD approach.
2010 and 2013. These results confirmed significant differences in
 Each test consists of 40 items. In each test all 13 chapters will
performance and pass rates between existing MCQ tests. They also
be covered by 3 MCQ’s: one testing knowledge (K) and two
demonstrated significant differences in the difficulty index (p
testing application of knowledge (A).
value) and the point biserial correlation (rpbi) values between
 The 40th question is used to address/emphasise specific
individual test items.
new topics in the 9th edition (for instance: balanced
Since changes in the content and format of the ATLS programme
resuscitation).
require evidence [5], the American College of Surgeons’ Interna-
2. Using the test matrix, all 160 questions from the initial tests
tional ATLS Subcommittee decided to install an International MCQ
were analysed and categorised. 45 items with a p-value
Revision group. The objective was to develop an updated and
(indicating the level of difficulty of a test item) below 0.25 or
balanced set of three equally difficult MCQ tests that would match
a negative discrimination index value (meaning that a test item
the content of the 9th edition ATLS manual and would meet the
does not adequately discriminate between good and bad overall
psychometric quality criteria. This paper describes the review and
test performers; also called rir-value, or rpbi-value) were
development process and its outcome.
directly excluded from future review. The quality of the
remaining 115 MCQs was assessed. 23 questions were classified
as good, 18 as moderate and 73 as questionable. After careful
Methods
review of the 73 questions of questionable quality, 49 questions
were excluded, based on a p-value > .90.
The International MCQ Revision group consisted of experienced
3. A subgroup from the MCQ Revision group critically reviewed the
medical doctors and educators that served also as ATLS faculty,
remaining 65 questions on content and mastery level (whether a
from different ATLS world regions and countries. The combination
MCQ tested K or A). Table 1 shows the numbers of questions that
of various countries and continents, backgrounds and languages,
were left unchanged and were adjusted.
was specifically sought to assure extensive expertise and input
Simultaneously, 78 new questions were formulated, most on
from around the world.
application. New MCQs were reviewed and adjusted in several
A development strategy and time frame were defined. The
rounds.
process was divided into seven steps that follow guidelines from
4. Combining new with reviewed questions resulted in a set of
Classical Test Theory (CTT) [6]:
143 well-constructed and peer-reviewed items. This set was
presented to the ATLS Subcommittee in March 2012 for final
1. Develop a test matrix (also known as a test blueprint or test
review.
specification matrix) [7] that addresses the learning objectives
and specifies each item at the intended mastery level:
knowledge (K) or application of knowledge (A).
2. Screen items from the 8th edition tests on re-usability, using Table 1
specific quality and content criteria. Match selected items with Results review process on pre-existing MCQs.
test matrix. N = 65 N per N available questions N of questions left
3. Review a subset of potentially reusable MCQs. Develop new quality on knowledge (K) and unchanged or
MCQs if necessary. Peer review of all questions by fellow-group level application (A) adjusted
members. Adjust MCQs. Good 23 K: 13, A: 10 Unchanged: 23
4. Present final draft MCQs to the ATLS Subcommittee and collect Moderate 18 K: 13, A: 5 Unchanged: 5,
feedback. adjusted: 13
Questionable 24 K: 13, A: 11 Unchanged: 8,
5. Refine questions. Beta-test a new product, in this case, the new
adjusted: 16
MCQs in a limited group of potential users and incorporate

Please cite this article in press as: Bustraan J, et al. MCQ tests in Advanced Trauma Life Support (ATLSß): Development and revision.
Injury (2015), http://dx.doi.org/10.1016/j.injury.2015.11.024
G Model
JINJ-6511; No. of Pages 4
J. Bustraan et al. / Injury, Int. J. Care Injured xxx (2015) xxx–xxx
5. All 143 MCQ’s were again critically judged and minor, mainly
textual, adjustments in a few questions were suggested and
implemented. Four new MCQs were formulated and added,
resulting in a set of 147 questions. These questions were beta-
tested with support of the ATLS Chicago Office in 661 ATLS
Instructors from across the world who answered the questions
on a secured website. Based on these results, 120 questions were
selected for the composition of three tests with equal
distribution of topic and mastery level of the question (K or
A). The three tests were then fielded for beta-testing, a process
supported by the ATLS Chicago Office. National ATLS Offices
invited their instructors to participate. Instructors logged in on a
website to take the test(s). Results were included in the analysis
only if all 40 questions had been answered.
Psychometric data showed that 21 items in one test (test 2)
had an rpbi value < .25, indicating these items to be poor
discriminators, whereas the other two tests only had one such
item. It was practically not feasible to test whether this
surprising outcome would be confirmed if test 2 would be
made by another group of instructors. Therefore, we decided to
invite the experts of the working group to critically review this
subset of poorly discriminating questions in test 2. We regarded
this expert validation as an acceptable approach to get this test
to an acceptable quality level. This step lead to only very few,
minor last changes.
6. As of November 2012, three tests were ready for use as part of
the 9th edition Update Process. Earlier ATLS updates showed
that countries needed some time after the new manual and
other course materials had become available to actually update
their instructors, course directors and coordinators. Therefore,
we assumed that the new tests would not be used before January
2013.
7. The performance of all three tests was scheduled to be re-
assessed in January 2014, based on global student test results.
However, as soon as countries started to work with the new
MCQ tests, it became apparent that the failure rate on MCQ
testing had increased significantly. Although reported pass/
failure rates varied per country and changed over time, a
persistent increased failure rate was noted, when compared to
the relatively stable success/failure rates in prior years.
Discussion
The ATLS Subcommittee took pre-existing concerns and
research outcomes on the quality of the test MCQs seriously by
installing an International MCQ Revision group as part of the 9th
edition Update Process. This stimulated the international dialogue
and cooperation within the ATLS community [10] on the tests,
which are important assets in maintaining ATLS Quality Standards.
The group succeeded in their task to develop three new tests for the
9th edition.
The application of rigorous evidence based test analysis and
development had the unintended consequence of lowering pass
rates for students of ATLS provide courses. This challenged the
reputation of the ATLS program. Possible explanations for
increased test difficulty were discussed. The MCQ Revision group
members, all having a high level of expertise in trauma care, may
have set the bar too high for provider course students while
revising the tests and test questions. The new questions were not
beta-tested among provider course students. Another possible
explanation might be that although protective measures were
taken, the 8th edition MCQ tests are accessible via the Internet.
This might have influenced earlier, positive MCQ test results.
Another explanation might be that in some courses the teaching
Please cite this article in press as: Bustraan J, et al. MCQ tests in Advanced Trauma Life Support (ATLSß): Development and revision.
Injury (2015), http://dx.doi.org/10.1016/j.injury.2015.11.024
3
might have been still following 8th edition where the tests were
already based on the 9th.
One might also have to reconsider whether MCQs are the best
option for testing the type of knowledge necessary to provide good
quality trauma care. Alternatives such as programmatic assess-
ment are described in recent assessment research as a tool to
achieve a coherent and consistent picture of student competence
[11,12].
The early performance results on the ATLS MCQ tests resulted in
several immediate changes to program delivery. The required pass
mark was modified to achieve historic pass/fail performance. ATLS
committed to beta testing items on students in future revisions.
Thresholds for test item difficulty and discrimination will require
further analysis in the future.
Despite careful process guidance, it again became clear that
test development projects are complex. Further research with
student data will be conducted and reveal more solid explana-
tions that can be accounted for in future revision projects. The
ATLS program benefitted from international efforts and leader-
ship to handle an unexpected and problematic situation ‘the ATLS
way’. Once the symptoms of a deteriorating situation became
obvious, the response was swift and damage was controlled. The
adjusted guidelines on student performance in the new tests
were discussed and agreed upon within the International ATLS
Community.
Authorship
JB conducted the literature search, study design and data
interpretation and wrote the initial manuscript. WH and IB
supported the study design and data interpretation and critically
revised the initial manuscript. KB and JK supported the data-
collection, analysis and interpretation and reviewed the manu-
script. JK critically revised the final manuscript. MH commented
the study design and supported data interpretation
Conflict of interest
The authors declare that there is no conflict of interest.
Acknowledgements
We acknowledge the contributions and support provided by the
Members of the ATLS International MCQ Working Group, the ATLS
Chicago Office and all instructors and course directors who
participated in the MCQ review and development process.
References
[1] Brasel KJ, Chapleau W, Al-khatib J, Haskin D, leBlanc P, Cardenas G, et al. ATLS:
the ninth edition. Subcommittee, American College of Surgeons Committee on
Trauma, International ATLS Working Group, Advanced Trauma Life Support
(ATLS). J Trauma Acute Care Surg 2013;74(5):1363–6.
[2] Mohammad A, Branicki F, Abu-Zidan FM. Educational and clinical impact of
Advanced Trauma Life Support (ATLS) courses: a systematic review. World J
Surg 2014;38:322–9.
[3] Schuwirth LWT, van der Vleuten CPM. Different written assessment methods:
what can be said about their strengths and weaknesses? Med Educ
2004;38:974–9.
[4] Palmer EJ, Devitt PG. Assessment of higher order cognitive skills in under-
graduate education: modified essay or multiple choice questions? Research
paper. BMC Med Educ 2007;7:49.
[5] Kortbeek JB, Al Turki SA, Ali J, Antoine JA, Bouillon B, Brasel KJ, et al. Advanced
Trauma Life Support, 8th edition, the evidence for change. J Trauma
2008;64(6):1638–50.
[6] Schuwirth WT, Vleuten CPM. General overview of the theories used in assess-
ment: AMEE Guide No. 57. Med Teach 2011;33:707–83.
[7] http://www.proftesting.com/test_topics/pdfs/steps_3.pdf.
[8] Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice-item
writing for classroom assessment. Appl Meas Educ 2002;15(3):309–34.
G Model
JINJ-6511; No. of Pages 4

4 J. Bustraan et al. / Injury, Int. J. Care Injured xxx (2015) xxx–xxx

[9] Lunz MA, Examination development guidelines. http://www.measurement-
research.com/media/evalguidelines.pdf. Publication date unknown.
[10] Kilroy DA. Teaching the trauma teachers: an international review of the
Advanced Trauma Life Support Instructor Course. Emerg Med J 2007;24:
467–70.
[11] Dijkstra J, Galbraith R, Hodges BD, McAvoy PA, McCrorie P, Southgate LJ, et al.
Expert validation of fit-for-purpose guidelines for designing programmes of
assessment. Med Educ 2012;12:20.
[12] Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S.
12 Tips for programmatic assessment. Med Teach 2014;1–6.

Please cite this article in press as: Bustraan J, et al. MCQ tests in Advanced Trauma Life Support (ATLSß): Development and revision.
Injury (2015), http://dx.doi.org/10.1016/j.injury.2015.11.024
View publication stats