See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/327015373

Assessing Multiple-Choice Questions (MCQs) and Structured Short-Answer

Questions (SSAQs) in Human Anatomy to Predict Students’ Examination
Performance

Article · January 2018

DOI: 10.5958/2231-6728.2018.00024.0

CITATION READS

1 6,590

3 authors:

Arun Dhakal Shekhar Yadav

Birat Medical College Chitwan Medical College
6 PUBLICATIONS   3 CITATIONS    6 PUBLICATIONS   2 CITATIONS   

SEE PROFILE SEE PROFILE

Govinda Prasad Dhungana

Tribhuvan University
37 PUBLICATIONS   81 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

research View project

[1] Malnutrition among under-5 children and associated factors- a community based cross-sectional study in Tanahun District of Nepal View project

All content following this page was uploaded by Govinda Prasad Dhungana on 20 August 2018.

The user has requested enhancement of the downloaded file.

 Journal of Research in Medical Education & Ethics DOI: 10.5958/2231-6728.2018.00024.0
Vol. 8, No. 2, July, 2018, pp-127-131

ORIGINAL RESEARCH

Assessing Multiple-Choice Questions (MCQs) and Structured Short-Answer

Questions (SSAQs) in Human Anatomy to Predict Students’ Examination
Performance
Arun Dhakal1*, Shekhar Kumar Yadav2 and Govinda Prasad Dhungana3
(*Corresponding author) email id: *arund@mail.ru, 2shekhar_np1@hotmail.com,
3
dhunganagovindana2012@gmail.com

ABSTRACT
Objectives: To evaluate multiple-choice questions (MCQs) and structured short-answer questions (SSAQs) items in human
anatomy by determining their discrimination and difficulty index, and performing ROC (receiver operating characteristics)
analysis to signify their ability to predict and discriminate poor and good performance of students. Methods: First and
second-year medical students of 2016 were followed through 1 year to register their internal examination performance (SSAQs
Downloaded From IP - 163.47.151.210 on dated 18-Aug-2018

and MCQs) of various items in anatomy. Responses were analysed and evaluated for performance. Results: A total of 30
SSAQs and 100 MCQs of five items were analysed. The difficulty index of MCQ and SSAQ was 0.36 and 0.38, respectively,
Members Copy, Not for Commercial Sale

which is considered optimal for both. SSAQ showed higher discrimination index (0.46 – considered very good item) than MCQ
www.IndianJournals.com

(0.29 – marginal item and subject to improvement), suggesting SSAQ a better tool to discriminate poor and good students than
MCQ. Conclusion: This study suggests that though difficulty index of SSAQ and MCQ are similar, and in terms of discriminating
good and poor students SSAQ was a better tool, ROC analysis revealed both the tools to be adequate in terms of discriminating
students with a relatively better value to SSAQ. As this study was conducted in a university where SSAQs are the only form
of evaluation of theoretical knowledge in anatomy, conducting similar study where both the tools are followed is highly
recommended.

Keywords: Predict, Assessment, MCQ, SSAQ, Human anatomy, Examination, Performance

INTRODUCTION assessment, namely MCQs, SAQs, Objective Structured

Practical Examination (OSPE) and Viva voce (VIVA),
Human anatomy is the foundation for professional
are commonly used to assess knowledge in anatomy of
training in the field of health sciences. As the subject is
undergraduate medical students. MCQs are reliable and
complex, teaching as well as evaluation of the subject
demands approach defined in relation to the subject valid and easy to administer to a large number of
itself. Methods implied for summative assessment of students[6,7]. Also, a large proportion of curriculum can
medical knowledge have predominantly included be tested in a single sitting, though construction of good
multiple-choice questions (MCQs) and essay-type MCQ is difficult and needs expertise. Structured short-
questions to evaluate recall of facts. The research shows answer questions (SSAQs) are questions on a particular
that assessment is the primary motivator and drive of topic where the question itself is structured for scoring
learning among students[1–3]. When tool of evaluation the student’s factual recall of knowledge together with
emphasises on recall of facts, students’ tend to adopt problem-solving ability. It has an advantage of giving
superficial approach to learning[4,5]. Various methods of students the liberty to construct their own answer rather
1
Assistant Professor, Department of Anatomy, Birat Medical College and Teaching Hospital, Biratnagar, Nepal
2
Assistant Professor, Department of Anatomy, Chitwan Medical College, Bharatpur, Nepal
3
Lecturer, Department of Statistics, Birendra Multiple Campus, Bharatpur, Nepal

IndianJournals.com 127
 Arun Dhakal, Shekhar Kumar Yadav and Govinda Prasad Dhungana
than choosing a set of preformed answers and yet
include sufficient precision for answers to be easily
marked[8]. Both the aforementioned tools of evaluation
need to be carefully constructed with provision of model
answers and training of the evaluators to avoid inter-
rater variability[3,9]. Furthermore assessment tools should
be valid, reliable and feasible and be able to measure
different facets of professional competencies.
The present article aims to evaluate the basis of validity
evidence of MCQs and SSAQs on human anatomy
employing difficult index, discrimination index, sensitivity,
specificity and areas under ROC curve.
MATERIALS AND METHODS
This study was done on 1st and 2nd-year medical
Downloaded From IP - 163.47.151.210 on dated 18-Aug-2018
undergraduates’ examination performance in the
discipline of human anatomy. Approval to conduct this
Members Copy, Not for Commercial Sale
www.IndianJournals.com
study was obtained from institutional review board of
Chitwan Medical College.
A total of 137 medical students of 1st year (n = 57) and
2nd year (n = 80) who were interested to volunteer for
the study were enrolled. SSAQs were selected from
past years’ final professional exam questions mixed with
newly evaluated questions. MCQs (A type) were
designed and discussed in the panel of anatomy faculties
for consistency.
Enrolled students appeared for 20 MCQs immediately
a day after sitting for their SSAQ (6 questions)-type
internal exams. As MCQ exhibited negative marking
for wrong answers, students were advised to respond
to confirmed known answers only. This avoided students
from guessing the answers without being penalised as
well as helped overcome bias in relation to SSAQ where
there is no liberty for presumptions.
As there are two internal assessments in 1st year and
three internal assessments in 2nd year, we got 137
students sitting for a total of 100 MCQs and 30 SSAQs
containing various systems of human anatomy.
Statistical analysis was performed using SPSS software
version 20. Difficulty index is defined as the percentage
128
of students who gets the item right, where the
percentage refers to the percentage of students from
the total of the lower and upper groups[10]. In this study,
the difficulty index of MCQ was determined by adding
the number of students who gave correct answer in the
upper and lower 27 per cent of students and dividing it
by total number of upper and lower 27 per cent of
students. Difficulty index for SSAQs was evaluated
similarly.
The discrimination index of an item is defined as the
degree to which it discriminates between students of
high and low achievement[10]. It can be used for weeding
out both too easy and too difficult items by maintaining
a target range of 20–80% [11]. Index of 0.40 and above
is regarded as very good items, 0.30 to 0.39 is reasonably
good, 0.20 to 0.29 needs improvement and 0.19 or below
is to be revised[12,13]. We calculated discrimination index
of MCQ by subtracting the number of students who
gave correct answer in the upper and lower 27 per cent
of students and dividing it by total number of upper and
lower 27 per cent of students. With similar approach,
the discrimination index of SSAQs is evaluated.
The area under ROC (receiver operating characteristics)
curve, sensitivity and specificity of SSAQ and MCQ
were achieved by ROC analysis. The curve values, more
than 0.70, were regarded as having a satisfactory
discriminative and predictive value, whereas more than
0.8 was considered having very good value[14].
RESULTS
A total of 57 1st year and 80 2nd year medical students’
SSAQ and MCQ marks were acquired from the
department of anatomy. Consent from department was
taken before collecting the data.
The difficulty index of MCQ and SSAQ was 0.36 and
0.38, respectively, which is considered optimal level of
difficulty for both [15,16] . SSAQ showed higher
discrimination index (0.46 – considered very good item)
than MCQ (0.29 – marginal item and subject to
improvement) [10, 13], suggesting SSAQs a better tool to
discriminate poor and good students in relation to MCQs
(Table 1).
Vol. 8, No. 2, July, 2018
 MCQs and SSAQs in Human Anatomy to Predict Students’ Examination Performance

Table 1: The difficulty and discrimination index of SSAQ reliability, feasibility and more [8,15,17–19]. This study
and MCQ in anatomy examination of medical students
examined MCQ and SSAQ for their effectiveness to
Index Assessment Tool
predict students’ performance.
MCQ SSAQ
Difficulty index 0.36 0.38 Our research outcome demonstrates that MCQ and
Discrimination index 0.29 0.46 SSAQ are efficient tools in terms of their difficulty and
ability to discriminate performance of poor and good
Table 2: The values of area under ROC curve, sensitivity and students. However, in relation to MCQs, SSAQs were
specificity of SSAQ and MCQ in anatomy examination of better able to discriminate students’ performance. There
medical students
may be various reasons implied behind the latter finding
Assessment ROC Sensitivity Specificity
Tool (CI 95%) (%) (%) in our case which is discussed. Assessment subjectivity
MCQ 0.85 (0.78–0.91) 76.3 84.2 is shown to influence the rating given by the
SSAQ 0.93 (0.88–0.96) 85.0 88.2 examiners[16,20]. As the issue of subjectivity is more
pertinent to SSAQs in respect to MCQs, to avoid it,
ROC analysis is shown in Table 2 and illustrated in Figure SSAQs were designed by three faculty members for
1. SSAQ with ROC value more than 0.9 and MCQ consistency in terms of rating the assessments. The
Downloaded From IP - 163.47.151.210 on dated 18-Aug-2018

with 0.8 reveals that MCQ is a good tool, whereas SSAQ university under which the institution (place of this
is an excellent tool of discriminating ability, research) functions, written form of assessment in basic
Members Copy, Not for Commercial Sale

respectively [14]. Also represented in Table 2 are the medical science uses structured essay-type questions
www.IndianJournals.com

sensitivity and specificity values of both the tools
indicating their good level to predict good and poor
students, though value of SSAQ is much better than
that of MCQ[14].
DISCUSSION
There are various studies interpreting tools of
assessment, their comparison, their source of validity,
Figure 1: Sensitivity and specificity of SSAQ and MCQ to
predict pass–fail outcomes
as the only form of evaluation. Also it must be noted
that these students of 1st and 2nd year have never faced
nor have any idea of MCQs in regards to basic medical
sciences. Students’ performance with MCQ also
showed low sensitivity. It is possible that university exams
acted as a motivating factor focusing students’ attention
on essay-type question where sudden exposure to MCQ
could have contributed to our present result though study
suggests that for most students, there is a strong
correlation between MCQ and Structured Essay
Questions (SEQ) indicating student performance
independent of testing format[21,22]. To elaborate the
reason further, locally question bank (SAQ type with
their answers) collected from previous years is rampant
in the market. As it is easier and more effective in terms
of scoring marks, students are drawn to those and are
reluctant to go through their recommended text books.
This approach has barred them from acquiring a broader
perspective of the subject and may also have reflected
in the result of our study.
With all the aforementioned potential explanations of
SSAQ being excellent to MCQ in our case we do not in
any way undermine MCQ as an effective tool in
discriminating performance of students. A well-

Journal of Research in Medical Education & Ethics 129

 Arun Dhakal, Shekhar Kumar Yadav and Govinda Prasad Dhungana
constructed MCQ is superior in testing the higher
cognitive skills of medical students [19]. There are
evidences to support MCQ’s validity to better
discriminate performance of poor and good students[23].
Moreover, when assessment tools are combined, formats
including MCQs were the best examination tools to
distinguish poor from moderate and excellent students[24].
Despite their advantages MCQs as assessment tool also
needs to be chosen particularly as test demands [25].
Considering aggregate performance MCQ papers
revealed to be less sensitive predictors than the essay
papers[26]. In addition, when comparable reliabilities of
a question are to be attained the effort of writing case
vignettes with SAQ items is less than writing MCQ
items[27].
Questions like SSAQs and modified essay questions
Downloaded From IP - 163.47.151.210 on dated 18-Aug-2018
(MEQs) test knowledge and evaluate problem-solving
skills together with benefit that the candidates build their
Members Copy, Not for Commercial Sale
www.IndianJournals.com
own answer rather than selecting from a list of answers
as that of an MCQ and yet support enough accuracy
for answers to be easily marked[8]. Also study evaluating
the results of examination using SAQ or MEQ
demonstrated significant correlation of both the questions
and concluded to minimise effort to construct MEQs by
introducing written examination consisting of both SAQ
and MEQ[18]. The main drawback of SAQ-type question
is that it is not easily computer marked and hence not
practical as an instrument for evaluating a large number
of students[17].
Few limitations of this study needs to be mentioned for
proper analysis and research prospect. As the study was
carried out on one institution, the results may not be
generalised and even seem out of context on other
setting. Variables like individual performance of students
and overall health of the students during the period of
taking test were not controlled.
This study suggests that though difficulty index of SSAQ
and MCQ are similar, and in terms of discriminating
good and poor students, SSAQ was a better tool, ROC
analysis revealed both the tools to be adequate in terms
of discriminating students with a relatively better value
to SSAQ. As this study was conducted in a university
130
where SSAQs are the only form of evaluation of
theoretical knowledge in anatomy, conducting similar
study where both the tools are followed is highly
recommended. Their own performance assessment with
any singular evaluation tool is not without any flaw. It is
indispensable that medical colleges take initiative and
devote time and attempt to assess their present
evaluation methods and develop guidelines on setting
up questions in alignment to learning needs.
REFERENCES
[1] Drew S. Student perceptions of what helps them learn and
develop in higher education. Teaching in Higher Education
2001;6(3):309–31.
[2] Trigwell K, Prosser M. Improving the quality of student
learning: the influence of learning context and student
approaches to learning on learning outcomes. Higher
Education 1991;22:251–66.
[3] Reid WA, Duvall E, Evans P. Relationship between
assessment results and approaches to learning and studying
in year two medical students. Medical Education
2007;41:754–62.
[4] Scouller K. The influence of assessment method on students’
learning approaches: multiple-choice question examination
versus assignment essay. Higher Education 1998;35:453–
72.
[5] Farley JK. The multiple choice test: writing the questions.
Nurse Educator 1989;14:10–2.
[6] Case S, Swanson D. Constructing written test questions for
the basic and clinical sciences. 3rd ed. Philadelphia: National
Board of Medical Examiners; 2003.
[7] Tarrant M, Ware JA. Framework for improving the quality
of multiple-choice Assessments. Nurse Educator
2012;37(3):98–104.
[8] Webber RH. Structured short-answer questions: an
alternative examination method. Medical Education
1992;26(1):58–62.
[9] Biggs J. Teaching for quality learning at university.
Philadelphia: The Society for Research into Higher
Education & Open University Press; 2003.
[10] Linn RL, Gronlund, NE. Measurement and assessment in
teaching. 7th ed. New Jersey: Prentice-Hall; 1995.
[11] Dixon RA. Evaluating and improving multiple-choice papers:
true-false questions in public health medicine. Medical
Education 1994;28(5):400–8.
Vol. 8, No. 2, July, 2018
 MCQs and SSAQs in Human Anatomy to Predict Students’ Examination Performance

[12] Rahim AFbA. What those number mean? 1st ed. Kubang
Kerian: KKMED; 2010. Available from: http://
www.medic.usm.my/dme/images/stories/staff/KKMED/
2010/item_analysis_guide.pdf.
[13] Ebel RL, Frisbie DA. Essentials of educational measurement.
5th ed. Englewood Cliffs. New Jersey: Prentice-Hall Inc;
1991.
[14] Thomas GT. Interpreting diagnostic test; 2009. Available
from: http://gim.unmc.edu/dxtests/Default.htm
[15] Edward JP, Peter GD. Assessment of higher order cognitive
skills in undergraduate education: modified essay or
multiple choice questions? BMC Medical Education
2007;7:49:10.
[16] Albanese MA. Challenges in using rater judgments in medical
education. Journal of Evaluation in Clinical Practice
2000;6(3):305–19.
[17] Sam AH, Hameed S, Harris J, Meeranr K. Validity of very
Downloaded From IP - 163.47.151.210 on dated 18-Aug-2018
[21] Pepple DJ, Laurine YE, Carroll RG. A comparison of student
performance in multiple-choice and long essay questions in
the MBBS stage I physiology examination at the University
of the West Indies (Mona Campus). Advances in Physiology
Education 2010;34(2):86–9.
[22] Mujeeb AM, Pardeshi ML, Ghongane BB. Comparative
assessment of multiple choice questions versus short essay
questions in pharmacology examinations. Indian Journal of
Medical Sciences 2010;64(3):118–24.
[23] Taib F, Yusoff MSB. Difficulty index, discrimination index,
sensitivity and specificity of long case and multiple choice
questions to predict medical students’ examination
performance. Journal of Taibah University Medical Sciences
2014;9(2):110–14.
[24] Eijsvogels TMH, van den Brand TL, Hopman MTE.
Multiple choice questions are superior to extended matching
questions to identify medicine and biomedical sciences
students who perform poorly. Perspectives on Medical

short answer versus single best answer questions for Education 2013;2(Nov(5–6)):252–63.
undergraduate assessment. BMC Medical Education
[25] Dascalua CG, Enachea AM, Mavrua RB, Zegan G.
Members Copy, Not for Commercial Sale

2016;16:266.
www.IndianJournals.com

Computer-based MCQ assessment for students in dental

[18] Wallerstedt S, Erickson G, Wallerstedt SM. Short answer medicine – advantages and drawbacks. Procedia – Social
questions or modified essay questions – more than a and Behavioral Sciences 2015;187:22–7.
technical issue. International Journal of Clinical Medicine
[26] Nnodim JO. Multiple-choice testing in anatomy. Medical
2012;3:28–30.
Education 1992;26(Jul(4)):301–9.
[19] Khan MuZ, Aljarallah BM. Evaluation of modified essay
[27] Rademakers J, Ten Cate TJ, Bär PR. Progress testing with
questions (MEQ) and multiple choice questions (MCQ) as
short answer questions. Medical Teacher 2005;27:578–82.
a tool for assessing the cognitive skills of undergraduate
medical students. International Journal of Health Sciences
Received: 20.02.2018
(Qassim) 2011;5(1):39–43.
Accepted: 10.06.2018
[20] Liao SC, Hunt EA, Walter C. Comparison between inter-
rater reliability and inter-rater agreement in performance
assessment. Annals of the Academy of Medicine, Singapore
2010;39:613–8.

Journal of Research in Medical Education & Ethics 131

View publication stats