Draft Research Outline

Title
Can multiple choice questions designed with Mayer’s principles of multimedia accelerate and
improve learning outcomes of vocal knowledge when used as content rather than
assessment?

Abstract
Multiple choice questions or MCQs display typical characteristics when used in
online learning packages. MCQs are always used as an instruments of assessment; and they
delivered after learner instruction is completed (as influenced by Gagne’s 9 items of
instruction). Further distinguishing MCQs are that their design is typically:
 a text question answered by selecting one or more possible answers also in text.
 Has a single question called a Stem, incorrect answers called Distractors and a correct
answer called a Key (Schimmelfing and Persky 2020).
While learner content that is aligned with the Cognitive Theory of Multimedia
learning and Mayer’s principles of multimedia acts to improve learner comprehension, MCQ
assessment is not aligned with CTML.
Can MCQ be used alternatively? Consider if a MCQ bank of questions was designed
with Mayer’s principles in mind?. For the purposes of identification, such a format is known
here as CMCQ (C representing CTML).
CMCQ derived assessment logically should optimise comprehension; making
assessment instruments clearer and with minimal extraneous cognitive load.
Consider further, as there is minimal extraneous cognitive load and clarity, could
positioning CMCQ questions as stimulus rather than assessment be maybe a shorthand way to
deliver learner content without verbosity? Can CMCQ be used to inform the learner of vocal
knowledge (facts) quicker than formal online lessons and promote good long term memory
retention?
Vocal knowledge as facts cannot be simply stated. Knowledge does require a
‘wrapper’ to allow learners context. For example, 6 x 6 =36 maybe a fact. But to learn this
fact some context is required to place this knowledge/scaffold it with prior understanding.
For example this could be contextualised to the learner as:
 In the 6 times table, 6 x 6 =36 (the ‘6 times table’ being the context)
A quick minimal text method to convey information is the multiple choice assessment format.
What may be three paragraphs of text as learner content can be abbreviated to one paragraph
in a MCQ.
Consider this learner content on the what are the tallest trees on Earth.
“The Coast redwood (Sequoia sempervirens) is the tallest tree species on earth. This is a list
of the tallest known tree species, as reflected by measurements of the tallest reliably-
measured individual specimen. Although giant trees grow in both tropical and temperate
regions, they are very restricted geographically and phylogenetically”. (from wikipedia)
Using a MCQ modeled as CMCQ this could instead be depicted to a learner as mostly image
and little text as follows.

What is the world’s tallest family of tree?

A/ Gumtree B/ Cyprus Pine C/ Redwood D/ Boab
 How does this CMCQ format act as learning? It is considered here that the action of
selection stimulates long term memory retention without the verbosity of typical multimedia.
Clearly, the learner may select incorrectly. However, the follow-up direction to
inform them of the fact that the Redwood is the tallest tree is the learning. Regardless of
selecting a right or wrong response, CTML theory limits extraneous cognitive load and
promotes retention; learners scaffold the correct response as a fact in memory and with less
effort and processing than the pure content format.
This concept then poses the following Research Question:
 is then CMCQ better and quicker to deliver learner outcomes than typical content
delivery: better learner retention and faster to develop/deliver?

Summary of this abstract - raison d’etre

1. Multiple choice questions (MCQ) typically look like this (source vanderbilt.edu)

2. MCQs are considered assessment tools and learners encounter them after content.
Gagne’s 9 items of instruction places them after feedback from the learner. The diagram
below depicts this. (source https://www.coursearc.com/gagnes-nine-events-of-instruction/)

3. Mayer’s principles of multimedia model makes comprehension of learning content easier

(source Introduction | The 12 Multimedia Instructional Principle)

4. Creating multiple choice questions that reference Mayer’s principles will also make
comprehension easier of a MCQ. Left is an aligned MCQ. Right is the MCQ unaligned.
Both ask the same question ‘what is the melt temperature of ice?’
5. When a MCQ becomes a CMCQ, it has a brevity of content to deliver facts fast and,
through the action of allowing learner selection, engages interest. Whether the key or
distractor answer is selected, re-inforcement of the correct answer stimulates cognition
and learning. A CMCQ can used as learning content as well as assessment for vocal
knowledge.

Five vocal facts as five CMCQ. A multimedia presentation

would require more content to deliver the same facts

A CMCQ as content maybe faster to produce than typical multimedia to achieve the
same learning outcome and maybe more successful as it is content stripped back to the
essential as opposed to typical multimedia (even if that multimedia is designed with Mayer’s
principles in mind.

Project Rationale - why this research matters

Considerable time is spent in adult education imparting low level vocal knowledge to
learners; vocal knowledge that is required as ‘bedrock’ prior learning before encountering of
higher order learning. ‘Higher order’ here is used in the Bloom Taxonomy sense as cognitive
activities beyond rote learned facts. If a quicker way could be found to pass on core facts on
a subject, more time could be spent on expansive imaginative thinking. And if quantitative
experimentation to examine CMCQ finds it is not quicker to develop/undertake, but equally
or more effective to typical multimedia in educating learners, CMCQ will add variety which
builds learner engagement as in the ‘attention’ sense in Keller’s ARC theory.
Based on CTML, it maybe possible to pass knowledge into short term then long term
memory using re-imagined multiple choice questions positioned as learning and not
assessment. Benajamin Wood developed what is considered the MCQ format in 1922. In
the 100 years since, MCQ has been a powerful instrument of assessment. Coupled with
CTML, it maybe that MCQ can be delivered earlier in the systematic instructional design life
cycle as learning.
note on research
 assessment is still required and it is not suggested formal testing can be ended;
assessment checks content comprehension.

Research Hypotheses
Research Question A
Does CMCQ improve learner retention compared to typical multimedia (Mayer styled)?
H0 Using CMCQ as learning does not enhance learner retention when compared to typical
multimedia based learning

H1 Using CMCQ as learning does enhance learner retention when compared to typical
multimedia based learning
‘Enhance’ as used above means a statistically better result understanding after assessment. An
enhanced outcome could be where learners exposed to CMCQ out perform non-CMCQ learners by
more than 10% as to total correct answers when assessed.

Research Question B
Is CMCQ faster to deliver than typical learning material where both formats have the same
ambition to teach the same vocal knowledge?
H0 Using CMCQ as learning is not quicker or is the same as typical learning material
H1 Using CMCQ as learning is quicker than typical learning material.
‘Quicker’ here means faster to create/deliver to students as online asynchronous learning in minutes
and seconds.
These research questions are fundamental. If CMCQ adds considerable time to the learning
process, it is not a viable instrument in content creation. Further, if CMCQ produces a lower
correct answer selection on assessment, it is not advisable to deliver learning content in this
fashion.

Methodology - Quantitative true experimentation post test only control group design

Overview
The research requires two hypotheses to be challenged. With detail provided in the
design and procedure sections below, the research will compare the results of two cohorts of
50 learners in each cohort. Required to learn the same facts, the variable distinguishing each
cohort is the format of the learner content.
Cohort A will be asked to parse a set of 50 questions designed as CMCQ queries.
Cohort B will undertake a MOOC course that has as an outcome the same desired vocal
knowledge understanding as the CMCQ led cohort A.
The same assessment (true/false, short answer, MCQ) will be taken by each cohort 7
to 14 days after the first learning phase.
Data analysis will then divine a response to the reference hypotheses (see above
H0/H1 for the two research questions). Of interest is how many and which questions did a
learner get correct/incorrect? And how long did the learners take to complete the initial
learning?

Participants
Adults aged 25 to 45 with entry skills to use competently a web browser. While there
is merit exploring K12 applicability with the research, access to a viable set of test candidates
will prove difficult in the time frame in which the research is to take place. Participants will
be randomly gathered from advertising for participants in coffee houses, gyms, supermarkets
in greater Penang. Participants do not have to meet the researcher and can do the research
online. Work needs to done BEFORE gathering data on how best to ensure participants in
both cohorts internal and external validity. (refer ‘problems with this research’ below).

Instruments
Cohort A - use Kahoot based multiple choice questions designed through the prism of
Mayer’s 12 principles of multimedia. Kahoot is a free browser based delivery of multiple
choice questions that allows simple creation of a CMCQ style format.

Cohort B - use a MOOC from the history teaching institute that uses an open format
over a web browser.
Design
The research is a quantitative true experimentation post test only control group design.
The one variable that two cohorts A and B consider is the content style they are exposed to.
Each cohort is 50 people strong. Cohort A undertakes a 50 question CMCQ format learning
package that covers the same vocal knowledge as a MOOC that cohort b completes.
The same assessment of comprehension using a traditional true/false, short answer,
MCQ (think non aligned with CTML) is undertaken by both cohorts after first completing
their respective learning modules.
Recording of results, time to complete and identification of the learner is recorded in
each cohorts instruments backend LMS.
Cohort A experiences a new format, while Cohort B experiences a typical multimedia
learning package.

Procedure
(note these elements run in many cases concurrently - research gantt chart to come will map the correlations.)

Produce research brief (grounded in this outline), final

literature review, hypotheses detail, experiment model

Design of learning Build a 50 question (estimated) CMCQ format that covers

more than 30 facts that a MOOC course in total includes
as a total fact dump. 50 so as to re-enforce correct
understanding - a fact noted the mooc course may have more
than one paired CMCQ question experience.

Formative assessment Review of CMCQ question set on Kahoot by multiple 3rd

parties to ensure ease of access, operation, recording of
results and clarity of purpose. Likewise, MOOC accessed
and similarly checked.

Gathering participants Early, a shout out to gather 100 participants will be

undertaken. Optimally 120 will be gathered so there
is reserve participants to counter mortality and statistical
Results outside 3 sigma variation (see research problems
section ‘statistics’ below.

Gathering dataset Complete 100 learner encounters with 100 matched

assessments; 50 cohort A and 50 cohort B that can be
passed as internal/externally valid.

Create assessment A true/false, short answer, multichoice assessment both

cohorts undertake 7 -14 days from exposure to learning
content.

Produce (II) Statistical evaluation, conclusions in relation to hypotheses.

Delivery 3 x hardcopy and 1 times soft copy APA formatted paper.

Data analysis
Descriptive statistics (mean, mode, median) will be created from a dataset where both
cohorts correct/incorrect responses to the post learning assessment will be produced. Of
interest, is the raw correct comparative scores, which questions demonstrate patterns of
correct or incorrect (if any), the time taken to complete the learning in total and by page turn.
Data outside 3 sigma variation will be discounted IF the number of responses outside 3 sigma
is not more than 5% of the total of 50 responses. Where the value is more than 5%,
consideration will be given to if the learning is poorly designed, there is a technical
instrument issue for example to explain the occurrence - discussion will be held with
experienced researchers as to if 5% is a reasonable extremity to exclude.

Time Schedule
12 weeks with an additional 6 weeks to data capture and analyse to report findings.
With in the initial 12 weeks, the core work (such as literature review) will be completed with
a final chapter of findings added to complete the research.

Early Literature review (sources researchgate.net)

There is considerable existing research into the application of CTML as a theory of
learning to empower learning outcomes. The root of work by Meyer and Moreno (2003)
congealed the earlier research of Sweller and Moreno (1998) into a working theory. CTML
has become arguably the most highly accepted theory on human learning. The application of
CTML led Mayer in 2009 to create a model (based on CTML) for the design of multimedia
which reduces cognitive load and assists learning. 12 principles in total, Mayer’s design
model likewise is recognised as integral to best practice in content creation.
While assessment is used to gauge if learners learn, there is little research on how
CTML could likewise benefit it; research into how CTML could play a role in the design of
assessment is ‘thin’. However, there is one primary research paper that this early stage
literature review uncovered: “Towards a cognitive theory of multimedia assessment
(CTMMA)” Kirschner, P.A, Park, B, Malone, S, & Jarodzka, H.
This research has as its core thrust that while multimedia content and CTML are well
explored, assessment in online learning over CTML is a combination that:

“lacks both theoretical underpinnings and practical design principles. Multimedia assessment
(MMA) is, at best, either a translation of paper‐based assessment and assessment principles to
the computer screen or an attempt to make use of the theory and principles underlying
multimedia learning (i.e., CTMML)” (Kirschner et al 2010).

The research explains the genesis of CTML and dual channel theory and explicitly
references Mayer’s principles. Table 1 over page is extracted from the research and is a
cross-reference of Mayer’s principles to assessment design. The format has a CTMMA
column that is the design model of assessments such as multiple choice questions. The
related CTMML column is the current use of Mayer’s principles in learner content.
Aside from the principle in Table 1 called ‘self-pacing’, 11 of 12 principles as
described by Mayer are mapped from learning to assessment. Self-pacing has some relation
to Keller’s ARC theory but this is not a direction for design. Where self-pacing originates
from is not documented.
By way of criticism it needs to be pointed out that the paper is a proposed model
without quantitative experimentation in support. Hence, no hypothesis is tested.
Secondly, the modality principle for assessment notes that pictures used in
assessment can ‘trick’ learners and so should be ‘cautiously’ used.
Using pictures to deliver meaning is a core plank of CTML and ‘scarce’ use makes
the application of CTMMA to assessment diminished; using a picture to re-frame a text
question where both have a strong correlation seems appropriate. In support of the claim,
Kirschner et al 2010 cite research into imagery by (Ogren and Nystrom 2016). True, the cited
reference work does note that a picture can dominate text and that care should be taken to
avoid drawing learner ‘attention and cognitive capacities away from focusing on the
statement’ (Ogren and Nystrom 2016).
 However, it does not claim that pictures should be used ‘scarcely’:

“when they were carefully framed and integrated with the problem statement they had a
beneficial effect on performance.”(Ogren and Nystrom 2016).

This seems at odds with the modality principle as described in table 1 and more in keeping
with Mayer’s direction on imagery in learning.

While there is little background to draw on with regard to CTML and assessment,
there is a substantive body of research into best outcome stem, distractor and key design;
although it is pertaining to text and not multimedia. There has been deep dives too into t he
logic of MCQ. Insight into leveraging prior knowledge has been explored by Schimmelfing
and Persky (2020) to further assist comprehension.

Summary of early stage literature review

By way of a summary, there is little research into making assessments work within
the bounds of CTML theory. There is no quantitative study. Of note, CTML ‘fueled’
multiple choice questions as learning before assessment is undocumented. However,
Searching for literature beyond Researchgate is required.
There is broad acceptance of CTML as a theory of information processing
and of Mayer’s 12 principles of design. But no study has been found that proposes to use
Mayer’s principles and/or to use a MCQ or CMCQ for that matter as a content to create vocal
knowledge in learners.

Table 1 of CTTML mapped to CTMMA from Kirschner et al (2010)

Principle CTMML CTMMA

Modality Pictures can actually ‘trick’ assessees into confirming a statement
People learn better from graphics and narrations than from
(cf. Ögren, Nyström, & Jarodzka, 2016). Hence, they should be
animation and on‐screen text, especially when the graphic is used scarcely and cautiously.

complex, the words is familiar, and the lesson is fast‐paced.

Segmentation
People learn better from a multimedia lesson is presented in
user‐paced segments rather than as a continuous unit.
Pre‐training
People learn better from a multimedia lesson when they know
the names and characteristics of the main concepts.
It is easier to distinguish between individuals of higher and
lower expertise, if the task or problem is presented as a
continuous unit (cf. whole task).
It is easier to distinguish between individuals of higher and lower
expertise, if no pre‐training on the test‐ material was given.

Coherence
People learn better when extraneous words, pictures and
sounds are excluded rather than included. Adding interesting
but irrelevant materials to e‐learning courses may distract the
learner.
Signaling People learn better when cues that highlight the organization of
the essential material are added.
It is easier to distinguish between individuals of higher and lower
expertise, if the amount of coherence of the testing material
corresponds to the coherence found in the real‐world.
It is easier to distinguish between individuals of higher and lower
expertise, if no additional cues or highlights
are given.

Spatial contiguity People learn better when corresponding words and pictures are
presented near rather than far from each other on the page or
screen.
It is easier to distinguish between individuals of higher and
lower expertise, if the spatial contiguity of the given material
corresponds to the real‐world situation: the assessment itself is
that the assessees select and
integrate the relevant information autonomously.
 Temporal People learn better when corresponding words and pictures are It is easier to distinguish between individuals of higher and lower
contiguity presented simultaneously rather than successively expertise, if all information is presented in such a way, as it would
occur in the real‐world task: for some situations this may mean
that people need to integrate a lot of information at the same time
or that they need to remember information for later usage.

Redundancy People learn better from graphics and narration than from It is easier to distinguish between individuals of higher and lower
graphics, narration and on‐screen text. The visual text information expertise, if the amount of redundant information is as high as it
presented simultaneously to the verbal information becomes would be in the according real‐world task or problem.
redundant.

Emotional Design and Emotion
Induction including
Personalization and Voice
People learn better from multimedia lessons when words are in
conversational style rather than formal style. People learn better
when the narration in multimedia lessons is spoken in a friendly
human voice rather than a machine voice.
Emotional design, emotion induction or personalization and
the use of human voice within the assessment situation could
help to distinguish between individuals of higher and lower
expertise because experts are known to be capable to
compensate effects of emotionalized material, induced
emotions or formal instead of conversational style.

Self‐pacing Learners learn better from self‐paced than from system‐paced As experts are known to execute tasks faster than novices do,
multimedia lessons putting temporal restrictions to assessment (presentation and
answer time) may help easier distinguish between individuals
of higher and lower expertise.

Problems with this research and follow-up research

It is acknowledged here that there exists issues with researching the proposition that
multiple choice questions designed with CTML in mind may be used as efficient learning
content with improved outcomes of retention.

Internal Validity/External validity

The research is undertaken as a quantitative true experimental post test only control
group design. To manage validity (threats such as history events, maturation of subjects, pre-
test sensitisation, unreliable instrumentation, mortality of participants, pre-test and multiple
treatment interference), there is:

 no pre-test or instructions before initial exposure to testing for either cohort

 a single expousre to the learning without repetition with a follow-up assessment 7 days
 Manipulation of treatments (random selection of participants and random assignment to
cohort A or cohort B).

Sample size
In this research, the research confidence target is 95%. All adults 25 to 45 years old
abd mentally capable of learning are within the audience of interest in considering CMCQ as
a viable method for learning of vocal knowledge. Being based on experimental quantitative
research, a sample size of 384 respondents is considered academically adequate to map to 300
million plus people (Krejcie & Morgan 1970) with a 95% surety.
It is noted that having 384 respondents is not a guarantee of success or, for that matter,
research failure of a hypothesis. Variation in location, cultural acclimation, age of learners,
sex and many other factors may impact a repetition of the research methods proposed used in
this study - the result will vary. The question that is germane is to what degree is that
variation?
The number of respondents is not proof per se. Repeated application of a CMCQ
model by researchers in a variety of settings is.
Primary research here to be conducted is not 384 subjects; it is a sample size of 100
adults between the ages of 25 and 45 years old. This age bracket contains people highly
likely to be digital content savvy and with sufficient entry skills to use the Kahoot application
(Kahoot is how results are gathered of CMCQ interactions) and to use a MOOC (an online
MOOC is the control groups learning platform).
With a sample size of 50 using the CMCQ instrument, the applicable population with
a confidence of 95% in the Krejcie & Morgan sense is low - no more than 44 people (oddly,
with Krejcie & Morgan’s oft used table, a 99% confidence with a sample of 50 has a higher
applicable population of 50 - 6 more than 95% confidence).
Many peer reviewed and published quantitative studies have sub 100 sample sizes.
The caveat being that these papers identify a need for a follow-up body of research in a
variety of settings with larger sample sizes.
The CMCQ model explored here is likewise requiring such a follow-up with many
large samples in a variety of settings being researched to say with high certainty CMCQ
promotes an alternate way to deliver vocal knowledge better and quicker.
What can be said from this research is that IF there is null or no improved speed of
delivery with null or no improved retention, then CMCQ at the least adds variety to the
delivery of learning content.
CMCQ is not a replacement for traditional content - it is a complementary route to
deliver learning.

Ethics
All research needs to be conducted within a framework of ethics. While the design of
this experiment aims to impart no psychological harm (for example, the CMCQ and MOOC
subjects are not controversial), full disclosure as to the ambition of the research could be
argued as ‘colouring’/influencing participants:

 by informing participants that the research is to assist in understanding if multiple choice

questions can be used to impart knowledge, will the control cohort try more or less hard?
Will the CMCQ cohort concentrate more?

The answer is unknown and careful consideration as to what is the brief provided to
participants.
This is not uncommon and all quantitative studies need management of participant
bias. In the proposed research noted here, participants will:

 not meet with the researcher (as in a one to one study).

 will not be close associates of the researcher.

As observed by Smith and Noble, “In quantitative studies, selection bias is often
reduced by the random selection of participants, and in the case of clinical trials
randomisation of participants into comparison groups” (Smith/Noble 2014). In this research,
a group of 100 people at arms length from the researcher will be collected that fall withing the
target audience (adult 25-45) and they will be computer random sorted into two cohorts of 50
people.
For a formal dissertation, any proposed research would first be required to pass an
institutions ethics committee; and the proposed ethical considerations here may or may not be
sufficient.

Statistics
Descriptive statistics will be used to gather mean, mode and median from the number
of correct answers and the alternate incorrect answers. As will statistics on the time taken to
complete the CMCQ and MOOC learning. Any statistics will be considered through the lens
of location. As noted by Labaree (2010), research results are setting contextualised because
‘…knowledge often takes a less rationalized and more informal character. Understandings
arise from the interaction among people, their work, and the contexts within which they live.”
Labaree (2010).
The results of this research are contemporised and inference can be drawn within
these boundaries. For example, while an inference can be made that classroom based learning
can benefit from CMCQ, further research is required within this setting.
 As recognised by Gay, Mills and Airasian (2012), quantitative research experiments
have to manage factors such as learner cheating, unclear test directions, too complex
vocabulary and poorly worded interactions.
To mitigate this, a formative assessment of all material will be made before gathering
data to ensure clarity. Also data gathered will be parsed through internal consistency
reliability.
Additionally, it is proposed to dispose of data that falls in a binomial distribution
outside 3 sigma variation (more than very poor or very good). Where results are discounted,
a fresh text subject will be added to keep the sample size as n=50.

Lack or peer reviewed research

While it is considered here that using CMCQ maybe a benefit as a technique, the lack
of prior research into assessment influenced by CTML practices is noteworthy. Further, using
an assessment technique as learning content in and of itself is unexplored.
Certainly it can be researched, quantitatively experimented on and explored. The
online application Kahoot allows for design of multiple choice questions compliant with
Mayer’s principles of design and CTML.
While early research, considering an assessment multiple choice tool as a learning
content creator maybe identifying a trend to come of creating a shorthand for learning
development.

Participants
Of significance is how to manage the validity of participants. Finding typical adults
aged 25 - 45 and ensuring their best contextual input is a challenging element. Consultation
with experienced researchers will be sought as a priority as to how to find and also counter
the impact on validity through participant actions.

Instrument reliability - MOOC/Kahoot

It maybe the case that the selected MOOC has an unreliable LMS feature and other
complications of access - it is a third party operation. Whereas Kahoot may have down time
that is unexpected. The instruments offer limited control to the researcher. If there is an issue
with these instruments, insight would be sought by research sponsors as to alternate
instruments.

References
Gay, Mills and Airasian , ‘Educational research - competencies for analysis and applications’ Pearson (2009)

Kirschner, P, Park, B, Malone, S, Jarodzka, ‘Towards a cognitive theory of multimedia assessment (ctmma)’,
Learning, Design, and Technology - An International Compendium of Theory, Research, Practice, and Policy
(2016)

Labaree.D, ‘The lure of statistics for educational researchers.’ Educational Research - the Ethics and Aesthetics of
Statistics (pp.13-25)

Ogram, Nystrom, Jarodzka, ‘There’s more to the multimedia effect that meets the eye’ Instructional Science (2017)

Schimmelfing.L, Persky.A, ‘Activating prior knowledge using multiple choice question distractors”, Medical
Education (2020)

Smith,Noble ‘Bias in Research. Evidence based nursing’ Queens University Belfast (2014)