An experience of evaluating designs of multimedia
educational materials for primary education
scar Navarro-Martnez
Equipo de Orientacin
CEIP San Jos de Calasanz
Tomelloso, Espaa
oscarnavarromartinez@gmail.com
Miguel Lacruz-Alcocer
Departamento de Pedagoga, Facultad de Educacin
Universidad de Castilla-La Mancha
13071-Paseo de la Universidad, 4. Ciudad Real
Miguel.Lacruz@uclm.es
AbstractThe increasing presence of multimedia materials in
the teaching and learning process demands greater attention paid
to their design and production. The purpose of this study is to
analyze and evaluate different formats for the presentation of
multimedia contents aimed to primary school pupils. The
ultimate goal is to establish a series of design guidelines when
preparing these kinds of resources. In order to do this, three
experiments were conducted using eye-tracking methodology and
then we compared two different formats of presentation and
tested which were the most appropriate settings. Upon further
analysis of the information collected in the three experiments, it
was concluded that a higher efficiency is achieved in learning
when less complex structures are used. Specifically, avoiding
distractions that do not provide relevant information and affect
negatively to a greater or lesser extent the learning process of the
students.
Keywordsmultimedia materials; eye tracking; evaluation;
empirical studies; design guidelines
I. INTRODUCTION
The important technological development that has occurred
in recent decades has led to a breakthrough in all areas. In
terms of education, it is becoming increasingly common to use
materials and multimedia resources as essential elements,
which are then integrated into the teaching and learning
process. The incorporation of elements such as interactive
whiteboards, netbooks, and Internet in our teaching activities
calls for an improvement in the development of multimedia
materials and resources. The future lines of work in the field of
education are transitioning from the traditional educational
materials (textbooks) to the incorporation of new media, such
as e-books and other materials in electronic format [1].
Our interest is centered on evaluating the different settings
and formats of presentation materials and educational resources
that are efficient and effective in the teaching process focusing
Ana Isabel Molina-Daz
Departamento de Tecnologas y Sistemas de Informacin,
Escuela Superior de Informtica de Ciudad Real
Universidad de Castilla-La Mancha
13071-Paseo de la Universidad, 4. Ciudad Real
AnaIsabel.Molina@uclm.es
particularly on primary education. For this purpose three
experiments have been conducted to learn about the reaction of
the pupils when exposed to the appearance of distracting
elements that can appear in a multimedia presentation. Such an
evaluation experience is intended to serve as a basis for the
subsequent definition of a series of multimedia educational
materials design guidelines.
The use of the technique of eye tracking is included in these
studies [2]. This technique allows us to record eye fixations, or
the stabilization of gaze points, that compose an individual
when viewing content displayed on the screen. From these
fixations we can calculate a series of metrics among which we
highlight the number of fixations and the time until the first
fixation on a certain part of the screen which we call area of
interest (AOI), as well as the number of fixations and the total
time of all fixations made. The AOIs are parts of the screen
containing the most relevant information in the texts and
images displayed. The several metrics from AOIs will enable
us to evaluate and compare different configurations of the
materials displayed. This technique has been used in various
areas, including advertising and evaluation of user interfaces
and web pages. Although there have been fewer studies related
directly to the field of education [4], the interest in this specific
field seems to be growing. The majority of existing research
focuses on the stages of secondary and higher education. While
it is scarce, we can also find recent publications carried out
with pupils of primary education and even with babies of only
a few months of life [3,4,5].
There are a number of works in which reference is made to
the Principles of Multimedia Learning by Richard Mayer
[6,7,8,4]. We will focus on evaluating three of these principles
in the study, which are used and described in this article. The
Signaling Principle refers to the fact that learners learn more
deeply when cues are added to highlight the organization of the
essential material. Second, the Coherence Principle, which
states that students learn better when extraneous and redundant
material is excluded rather than included from the presentation.
Finally, the Redundancy Principle, which states that students
learn better from animation and narration than from animation,
narration, and text. Our study can also be related to the Gestalt
Principles, specifically the Law of Pragnanz, which proposes
that reality is organized or reduced to the simplest form
possible [9].
There are several previous studies that claim that redundant
information diverts the attention of university students and
therefore hinders their learning and academic performance
[10,11]. Some of the previous studies that include the use of
sound (audio) to promote the assimilation of contents by
students and even more at the age considered in this work (7 to
11 years old). There are studies dealing with the acquisition of
knowledge through multiple channels which use descriptions
by means of audio files with students of similar age [12] and
even the replacement of written texts by others spoken to
contrast their effectiveness [13,14]. The evaluation of materials
that include texts and images, both together and separately, to
improve the learning process has also been studied using eye
tracking techniques. For example, from studies of Richard
Mayer with a more theoretical approach [7,8], to others that
corroborate these principles with students of similar
educational stage [3], subsequent [15,16], and even preschool
[17].
Another element in that this research aims to investigate is
the use of color in multimedia. According to some authors, that
deepen on usability in a multimedia presentation it is important
to use colors that are easy to differentiate between and use a
high contrast to distinguish them by saturation, brightness and
tone [9,18]. Primary and warm colors are recommended for use
in materials designed for children [19]. There are other studies
with adults which used eye tracking techniques that analyze the
possibilities of the use of color to promote the search for
relevant information from text and/or illustrations [2,20].
II. METHOD
This research was conducted at Castilla-La Mancha, in
Spain. The school board and tutors of the second and sixth
grade of primary education collaborated in this study. These
two groups were chosen since they represent two key
evolutionary moments of the primary education student with
very pronounced differences, from the developmental and
psycho-evolutionary point of view, which are not so delimited
in other grades. We have avoided first grade since the linguistic
skills of the pupils may vary too much, while at second grade
more linguistic uniformity exists.
This study intends to analyze and assess the presentation of
multimedia content in different formats with students at the
stage of primary education. The general hypotheses that are
intended to contrast are:
H0: There is no greater efficiency in content retention if
these occur with absence of disruptive elements and
distracters.
H1: There is a greater efficiency in content retention if these
occur with absence of disruptive elements and distracters.
The experimental methodological approach will be similar
in all three studies. We will give a brief description of the
common schema followed in the three experiments and then
explain the specifics of each one individually.
A. Participants
The participants consisted of 49 primary education
students, 25 in the second grade (M = 7.55, SD = 0.29) and 24
in the sixth grade (M = 11.72, SD = 0.41), all Spaniards or
children of immigrants in school from the pre-primary stage.
A pilot test with 8 students was carried out to adjust the
times of the presentations so they reflected the appropriate
duration. Also, there were several calibration tests of the eye
tracking device (eye tracker) with students from the second
year of primary education, since typically these students
present a more dispersed attention.
B. Materials
The Felder-Silverman Learning Style Model (Felder Test)
was completed for the formation of the groups since it allows
to identify the style of learning [21]. The Kaufman Brief
Intelligence Test (K-BIT) [22] was also used, in this study to
calculate the Intelligence quotient (IQ). In addition, a
questionnaire with open ended questions was completed to find
out the prior knowledge of the students.
The materials presented to the students consisted of
presentations with images, text and sound produced with
different software (Adobe Flash CS5, PowerPoint 2007, Gimp,
Sound Forge 8.0b and Tobii Studio 3.0.5.301). The material
that was required for every experiment and hypothesis to be
verified was composed of different formats, including texts and
oral narration in Spanish as well as images.
At the end of the presentation, the participants completed
an activity intended to determine the level of assimilation of
the contents displayed. These contents are further explained in
annex II of the Royal Decree 1513/2006 of 7 December, which
establishes the minimum teaching of primary education. The
contents belong to the block of contents number three in the
area of Mathematics (Geometry), which refers to "the
identification and description of flat shapes, its elements, as
well as the situation in terms of angles and spins, and the
recognition of symmetries and regularities" [23]. Other studies
focused on the area of Mathematics [1,24] can be found,
including some which make explicit references to the
geometric figures [2].
C. Equipment and spaces
This study was executed in four units of the educational
center. A Tobii device model X60 was used for the eye
tracking, which includes specific software (Tobii Studio
3.0.5.301) required for calibration, the design of the
presentation, the collection of data and the subsequent
calculation of the different metrics.
D. Experimental design
Figure 1 consists of a graph detailing the process followed
in the first component of the experiences whose schema is very
similar to that used in the three experiments.
For sampling, the technique used was the method of Quota
Sampling [25] since the individuals in the population were
previously known. Characteristics from which this process is
carried out are IQ and learning style. The first characteristic is
defined with the K-BIT.
E. Experiment 1
The first experiment focuses on the evaluation of the use of
color in a presentation of images and text. The experimental
group (Group 2) observed an image with a combination of
colors that make a good contrast, which favors the perception
and visualization of the contents by the student [9,18]. Primary
and warm colors have been used since they are preferable to
create content aimed at children than pastels, such as those
presented for the control group (Group 1) [19].
Aforementioned and based on the bibliography of Richard
Mayer, the Signaling Principle, which refers to the need to add
signals that highlight the organization of relevant elements, has
been taken into consideration.We also tried to consider
situations that tend to the simplicity and clarity to promote
learning (Law of Pragnanz).
The null and alternative hypothesis intended to test in this
first study are as follows:
H0: There is no greater efficiency in content retention if we
include a presentation format with colors that make good
contrast.
H1: There is greater efficiency in content retention if we
include a presentation format with colors that make good
contrast.
The block of content referred to in the materials displayed
in both grades is the same, "interpretation and representation of
shapes and situation in the space". However, these contents are
different in nature due to problems that emerged in the data
collection for 6th grade of primary education and therefore, it
was necessary to replace the initial presentation.
F. Experiment 2
Text has been included in the presentation of the second
practice. In both cases it incorporates an identical oral narration
(audio). Let us note that only images and oral narration appear
for the experimental group (Group 2).
In contrast, for the control group (Group 1) an extra
element was added, a written text displaying the same words
heard in the audio recording.
Referring again to the biography of Richard Mayer, the
Redundancy Principle will also be considered. According to
this principle, the use of images and oral narration (audio)
improves the learning process in a greater extent than if you
also include written text.
The null and alternative hypothesis intended to test in this
study are as follows:
H0: There is no greater efficiency in content retention if
these are presented with pictures and oral narration only rather
than images, oral narration and text writing.
H1: There is greater efficiency in content retention if these
are presented with pictures and oral narration only rather than
images, oral narration and text writing.
The contents used were the same in both groups except for
a different complexity level due to the age of the students.
Larger images and font size were used for second grade (2 nd)
while longer texts and smaller font size were used for sixth
grade (6th) since they have completed four more academic
years in the primary stage and have more developed language
skills.
G. Experiment 3
This experiment includes the emergence of foreign
elements as a variable, such as superfluous images on the edges
and sounds corresponding to the appearance of the next slide.
These superfluous images appear for the experimental group
(Group 1) while they do not appear for the control group
(Group 2).
The Coherence Principle applies here as it says that a
multimedia presentation promotes learning if it avoids the
appearance of foreign elements. Similarly, we can also point
out the Law of Pragnanz, in terms of finding simple ways
without the use of disruptive elements that may distort or divert
attention, such as sounds or images that do not provide new
information on the edges of the slide.
The null and alternative hypothesis intended to test in this
study are as follows:
H0: There is no greater efficiency in content retention if
these are presented without disruptive elements, instead of
superfluous images and sounds.
H1: There is greater efficiency in content retention if these
are presented without disruptive elements, instead of
superfluous images and sounds.
 III. RESULTS
As shown in Figure 2, the results of the post-test and total
time display of content for each of the groups in Experiment 1.
Group 1 represents the students presented with figures that are
more difficult to distinguish, compared to those in Group 2 in
which we used a color-coding with a better contrast. We can
observe that the score of the post-test is greater for Group 2 in
differences were found for the results of the post-test (t = 0.91
and p = 0.19). This is not the case for the time of display,
where we found significant differences in t = 1.73 and p = 0.05.
In terms of time to the first fixation, it cannot be said that there
are significant differences because the values calculated are
lower than the critical value. On the contrary, the results for the
second grade show the total viewing time (t = 2.098 and p =
0.024) and the total number of fixations (t = 2.03 and p =
0.028) set values in the AOIs to affirm that there are significant
differences. Group 2 displays the AOIs during less time and
fewer attempts.
In Experiment 2, the students watch a presentation that
includes images, oral narration, and extra written text in the
case of Group 1. Figure 3 shows that the results of the post-test
in both groups are very similar. Where clear differences are
detected is in the total duration of the mountings generated. We
found significant differences in the same metrics and in both
groups (2nd and 6th grade). We refer first to the total time for
second grade, whose value is considerably shorter for the
Group 2, t = 2.82 (p = 0.007). Likewise, the total time duration
of Group 2 (6th grade) is shorter but in a lesser extent, with a
value of t = 1.79 (p = 0.045). In terms of the eye tracker
metrics, we can affirm the AOIs in Group 2 are displayed quite
earlier, since the values obtained are significantly greater for
Group 1 in the time that elapses until the first fixation (t = 3.99
and 0.001 to 2nd grade, t = 3.67, and 0.001 to 6th grade), as well
as in the number of fixations until the first one in the AOI (t =
3.75 and 0.002 for second grade, t = 4.67 and p < 0.001 for 6th
grade). We also find significant differences in the total time of
visualization (t = 3.97 and p < 0.001 for 2nd grade, t = 4.43 and
p < 0.001 for 6th grade) and in the number of fixations (t = 5.73
and p < 0.001 for 2nd grade, t = 5.70 and p < 0.001 for 6th
grade), maintaining the gaze for a shorter time in the case of
Group 2, and generating a smaller number of fixations.

comparison of the two grades, but it is also achieved in less

time as the duration of the visualization by the student is longer
in Group 2. Using the value calculated by the Student's t-test (t-
test) for the post-test in second grade, with a significance level
of 0.05, we observe a value of t = 1.92 (p = 0.037) which is,
greater than the critical value. Therefore, our findings express
significant differences between the samples from Group 1 and
Group 2. This allows us to assert that the post-test score for
Group 2 is significantly higher. Additionally, regarding to the
duration of the presentation, the value of t = 0.88 (p = 0.39) is
less than the critical value, and we cannot consider that there
are significant differences between both groups. With regard to
the metrics provided by the eye tracker, we will focus on four
of the metrics for each AOI (number and time until the first
fixation as well as number and total time of all the fixations).
In the case of Group 2 the fixation in the AOI is performed
before (t = 2.29, p = 0.017). In addition, we also note that the
total time of display, calculated from the total time of the
fixations generated, the AOI (t = 1.98, p = 0.032) and the
number of fixations (t = 1.84 , p = 0.043 ) is less for Group 2,
although the lower time and number of fixations allow for a
higher score in the post-test. With regards to the students of
sixth grade, expressed in Figure 2, the results are similar to
those obtained by the students of second grade; with a higher
score in less time for the Group 2. However, no significant
 Finally, in the case of the materials displayed in
Experiment 3, we show some foreign elements that trouble the
perception of boys and girls. Each time a new slide appears
superfluous sounds are heard and unnecessary images are
displayed on the edges that can distract your attention. It is in
this experiment where we find a greater difference between the
two grades. As we observe in Figure 4, the total time of display
is shorter for both cases in the Group 1. The score of the post-
test is very similar for second grade and higher in Group 1 for
sixth grade. Firstly, it should be noted that for second grade we
cannot conclude that there are significant differences in neither
the results of the post-test (t = 0.072 and p = 0.47) nor in the
total duration of the display (t = 0.60 and p = 0.28). We can say
that students fix their attention on the AOIs earlier in the case
of Group 1, considering both the time until the first fixation (t
= 4.15 and p < 0.001) and the number of fixations (t = 4.01 and
p < 0.001) before fixing the gaze at an AOI. Additionally,
when reviewing the results of sixth grade students we find
significant differences in the results of the post-test (t = 1.95
and p = 0.035) but not in time (t = 1.32 and p = 0.10).
According to the metrics of the eye tracker and in the case of
second grade students, we note that the gaze is turned to the
AOIs much earlier in Group 1 as per both the time until the
first fixation (t = 6.33 and p < 0.001 ) and the number of
fixations (t = 7.25 and p < 0.001 ). Moreover, the total time of
fixation into the AOIs is lower (t = 2 and p = 0.031) and a
smaller number of fixations take place in Group 1 (t = 2.4 and
p = 0.014).
IV. DISCUSSION
Analyzing the values obtained for second and sixth grade in
Experiment 1 with a level of statistical significance of 0.05, we
can assert that we do not accept the null hypothesis and accept
the alternative hypothesis for both educational grades, which
indicates that there is a greater efficiency in the assimilation of
the contents when using color-coding that make better contrast
in the display therefore favoring the visual perception of the
main elements displayed. In both cases second and sixth grade
obtained a higher score and a shorter time in the post-test. This
efficiency is also reflected in the metrics provided by the eye
tracker due to the fact that the AOIs are displayed fewer times
and for a shorter period, offering favorable results for the
experimental group (Group 2). In addition, we can conclude
that the first visualization in an AOI is performed in less time
in regards to second grade.
In Experiment 2, we also accept the alternative hypothesis
confirming that there is a greater efficiency in the retention of
educational content if it is presented with images and oral
narration instead of images, oral narration and extra text on
screen. The score of the post-test is very similar but in less total
time. In addition to this, the gaze is turned much earlier toward
the AOIs in the case of students of the experimental group
(Group 2). Also, results in this group show there are fewer
fixations into the AOIs and in less time, for both second and
sixth grade.
For Experiment 3, the alternative hypothesis is validated
only in the case of sixth grade. There is a higher efficiency in
learning because there is a higher score in the post-test in a
similar time. Therefore, assimilation of multimedia contents is
promoted when superfluous images and sounds do not appear.
However, we cannot assert that there is a significant difference
in the case of second grade. We can state that the fixation on
the AOI appears earlier in both cases and involves a fewer
number of fixations but with less total duration in those areas.
In most cases, the appearance of distracting elements, for
example, inappropriate colors, exposure to unnecessary text,
and redundant images or sounds, is prejudicial to the
assimilation of multimedia content and materials, therefore
reducing its efficiency. Moreover, a clear trend to gaze a
smaller number of times and earlier in the areas that contain the
key elements (AOIs) is observed when these distracting
elements do not appear. Furthermore, the total duration of
content fixations is lower in the experimental group, although
of a sufficient duration to assimilate the content.
V. CONCLUSSION AND SUGGESTION FOR FUTURE STUDIES
The Principles of Mayer and other authors on the subject
are corroborated in this study. However, this study is limited to
a sample of students of primary education (between 6 and 12
years) and there is little research in this stage of education
compared to that of secondary school or university.
Taking into account the results obtained, we can begin
shaping some patterns of design and development of
multimedia educational materials for primary education. Let us
note that the guidelines below have been tested objectively by
using an eye tracker device. These guidelines are:
 We recommend using primary and warm colors with all the
students instead of the pastel colors, which are not
recommended. An appropriate contrast of colors allows the
students to differentiate easily between images and texts.
We recommend avoiding the inclusion of redundant or
repetitive written text, especially with younger pupils (second
grade). Older pupils of primary school (sixth grade) show
similar results even though they spend a shorter time
visualizing the text.
We recommend avoiding the inclusion of foreign elements
such as redundant sounds and unnecessary images especially
with older pupils due to the fact that at that age they are paying
more attention to these elements. Younger pupils provide very
similar results.
In future lines of investigation we consider studying the
affect of presentations of multimedia content with the
emergence of other distracting elements, such as pop-ups or the
position of the AOIs on screen.
ACKNOWLEDGMENT
The authors would like to thank Grupo CHICO of the
UCLM (University of Castilla La-Mancha) for facilitating the
use of the Tobii X60 device, as well as the school board, tutors,
and pupils of second and sixth grade from CEIP San Jos de
Calasanz for having participated in this research.
REFERENCES
[1] Andr C., Arzarello F., Ferrara F., Holmqvist K., Lindstrm P., Robutti
O., Sabena C., How students read mathematical representa-tions: an
eye tracking study, Proceedings of the 33rd Conference of the
Intern.Group Psyc. of Math. Educ., Vol. 2, pp. 49-56, 2009.
[2] Garca-Hernndez C., Eye Tracking technology applied to the design of
safety and health signs at work, ORP, A Corua, 2008.
[3] Mason L., Tornatora M.C., Pluchino P., Do fourth graders integrate
text and picture in processing and learning from an illustrated science
text? Evidence from eye-movements patterns, Computers and
Education, Vol. 60, pp. 95-109, 2013. .
[4] Mayer, R.E., Unique contributions of eye-tracking research to the study
of learning with grafics, Learning and Instruction, Vol. 20, pp. 167-
171, 2010. .
[5] Sim G., Cassidy B., Read J.C., Understanding the fidelity effect when
evaluating games with children, Proceedings of the 12th Int.
Conference on Interaction Design and Children, pp. 193-200, 2013.
[6] Mayer R.E., The Cambridge handbook of multimedia learning,
Cambridge University Press, 2005.
[7] Mayer R.E., The promise of multimedia learning: using the same
instructional design methods across different media, Learning and
Instruction, Vol. 13, pp. 125-139, 2003.
[8] Mayer R.E.,Multimedia aids to problem-solving transfer, Interna-
tional Journal of Educational Research, Vol. 31, pp. 611-623, 1999.
[9] Ware C., "Visual thinking for design", Burlington: Morgan Kaufmann,
pp. 65-85, 2008.
[10] Liu H., Lai M., Chuang H., Using Eye-tracking technology to
investigate the redundant effect of multimedia web page son viewers
cognitive processes, Computers in Human Behavior, Vol. 27, pp. 2410-
2417, 2011.
[11] Van Gog T., Jarodzka H., Scheiter K., Gerjets P., Paas F., Attention
guidance during example study via the models eye movements,
Computers in Human Behavior, Vol. 25, pp. 785-791, 2009.
[12] Krejtz I., Szarkowska A., Krejtz D., Walczak A., Duchowski A., Audio
description as an aural guide of childrens visual attention: evidence
from an Eye-tracking study, Proceedings of the Sympo-sium on Eye
Tracking Research and Applications, pp. 99-106, 2012.
[13] Van Gog T., Scheiter K., Eye tracking as a tool to study and enhance
multimedia learning, Learning & Instruction,Vol.20, pp.95-99, 2010.
[14] Wiebe E.N. Anneta L.A., Animation and narration: using Eye Tracking
to understand visual attention distribution, Annual meeting of the
American Educational Research Association, Chicago, 2007.
[15] Mason L., Pluchino P., Tornatora M.C., N. Ariasi, An Eye-tracking
study of learning from science text with concrete and abstract
illustrations, The Journal of Experimental Education, Vol. 81, pp. 356-
384, 2013.
[16] Yang, F., Chang, C., Chien, W., Chien, Y., Tseng, Y., Tracking
learners visual attention during a multimedia presentation in a real
classroom, Computers and Education, Vol. 62, pp. 208-220, 2012.
[17] Evans M.A., Saint-Aubin J., What children are looking at during shared
storybook reading: evidence from eye movement monitoring,
Psychological Science, Vol. 16, pp. 913-920, 2005.
[18] Johnson, J., Designing with the mind in mind: simple guide to
understanding user interface design rules, Morgan Kaufmann, pp. 53-
63, 2010.
[19] Romero, G., Pautas de Diseo de Interfaces Grficas Basadas en el
Modelo de Aprendizaje S.O.I., Plataformas: Microsoft, Linux.,
Docencia Universitaria, Vol. IV, pp. 71, 2003.
[20] Ozcelik E., Karakus T., Kursun E., Cagiltay K., An Eye-tracking study
of how color coding affects multimedia learning, Computers and
Education, Vol. 53, pp. 445-453, 2009.
[21] Felder, R.M., Silverman, L.K., Learning and Teaching Styles in
Engineering Education Application, Engr. Education, Vol. 78, pp. 674-
681, 1988.
[22] Kaufman, A.S., Kaufman, N.L., K-BIT: test breve de inteligencia de
Kaufman, Tea, 2000.
[23] Real Decreto 1513/2006, de 7 de diciembre, por el que se establecen las
enseanzas mnimas de la Educacin primaria.
[24] Moreno R., Durn R., Do multiple representations need explanations?
The role of verbal guidance and individual differences in multimedia
mathematics learning, Journal of Educational Psychology, Vol. 96, pp.
492-503, 2004.
[25] Kish, L., Muestreo de Encuestas, Trillas, Mxico, 1975