22 (IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 9, September 2010
Evaluating Learning Objects for Students: An
adaptive model
Sonal Chawla and Dr. R.K. Singla
Dept of Computer Science & Appl, Panjab
University, Chandigarh. INDIA
sonal_chawla@yahoo.com
Abstract: The research paper looks at the perception of
students and their readiness for the new approach towards
learning through the Learning Objects. The Learning Objects
for the C++ course were developed as part of this study and then
tested using the LORI scale and the performance of the students
taught using learning objects was compared with the students
learning the same course in the traditional way.Finally the
conclusions are drawn.
Keywords: Learning Objects, LORI Scoresheets, e-learning,
learning evaluation.
1. Introduction
The development of effective content suiting the learning
style of users and the prevailing learning scenarios improves
the success rate of an e-Learning initiative significantly. It
is, therefore, important for content to adhere to the
objectives of the program and be powerful enough to engage
the user. The establishment of means of quality assurance,
requires criteria for evaluation that supports the
communication of meaningful feedback to designers for
content information.
Development of Learning objects that matches intended
outcomes and delivers the requisite cognitive load requires
careful planning and structured development. For that
purpose, Nesbit and Li [10] developed a Learning Object
Review Instrument (LORI 1.5) which can be used to reliably
assess some aspects of Learning Objects. This approach was
adopted in the design of their convergent participation
model for the evaluation of learning objects. Their model
proposed an evaluation panel drawn from different
stakeholder groups and a two-cycle process, whereby
participants would begin by evaluating the learning object
independently and asynchronously. The two-stage cycle was
facilitated by electronic communication tools and used the
Learning Object Review Instrument (LORI) to specify the
rating scale and criteria of evaluation. Subsequent research
on the use of the LORI revealed that objects that were
evaluated collaboratively led to greater inter-rater reliability
as opposed to ones evaluated independently
There have been a limited number of empirical studies
examining the learning outcomes and the instructional
effectiveness of Learning Objects despite the fact that
Learning Object repositories commonly use the review
instruments. A few of the worth mentioning are the study by
Kay and Knaack [3][2] which examines the quality of
Learning Objects through content analysis of open-ended
response questions based on principles of instructional
design and perceived benefit under post-hoc structured
categories. They evaluated five learning objects with
secondary school students but this study has the limitation
that it focused only on perceived benefits of Learning
Objects rather than on the actual learning outcomes
resulting from the Learning Object activities.
In another study Akpinar and Simsek [1] tested eight
Learning Objects, with school children in a pre-post test
research design. The data analysis revealed that seven of the
Learning Objects helped the sample students improve their
pretest scores, but in one, the Horizontal Projectile Motion
(HPM) LO for ninth grade students, the scores did not
improve.
On similar design, Nurmi and Jaakkola [4] conducted an
experimental study using a pre-test post-test design to
evaluate the effectiveness of three Learning Objects from
three different subject areas, i.e. Mathematics, Finnish
Language and Science. The Learning Objects, tested with
school children, were used in different instructional settings.
The results revealed that no significant differences were
observed between the Learning Object and the traditional
teaching conditions with low and high prior knowledge
students.
This study, thus, developed ten Learning Objects for the
C++ course using the Authoring Software ‘Xerte’ [9] and
‘Moodle’ and then tested them by conducting two studies.
The details of Learning Object development have been
deliberately kept out of this research paper since its major
focus is on evaluation of developed Learning Objects.
2. Study 1: To evaluate the Quality of Learning
Objects using the LORI score sheets.
2.1 Aim of the Study
To evaluate the quality of developed Learning Objects the
students were asked to rate and review the Learning objects
individually using the LORI score sheets (Sample sheet
variables discussed below). Following the reviewing and the
rating process, the researcher combined the ratings and
estimated average rating for each Learning Object. Average
ratings were estimated both for each of nine issues for a
particular Learning Object and for the overall rating of that
Learning Object.
 (IJCNS) International Journal of Computer and Network Security, 23
Vol. 2, No. 9, September 2010

2.2 Methodology Interaction Usability (V6): refers to ease of

navigation, predictability of the user interface and
The evaluation of Learning Objects for their quality was
quality of the interface help features.
based on the LORI (Learning Object Review Instrument)
scale [10]. Dimensions for evaluating learning object quality
in LORI 1.5 have been listed below:-
Content Quality (V1): refers to Veracity, accuracy,
Accessibility (V7): refers to Design of controls and
balanced presentation of ideas, and appropriate level of
presentation formats to accommodate disabled and
detail included in Learning Objects.
mobile learners
Learning Goal Alignment (V2): refers to Alignment
Reusability (V8): refers to the ability to use in varying
among learning goals, activities, assessments, and
learning contexts and with learners from differing
learner characteristics.
backgrounds.
Feedback and Adaptation (V3): refers to Adaptive
Standards Compliance (V9): refers to the adherence to
content or feedback driven by differential learner input
international standards and specifications.
or learner modeling.
Motivation (V4): refers to the ability to motivate and
interest an identified population of learners.
Presentation Design (V5): refers to the design of visual
and auditory information for enhanced learning and
efficient mental processing.

2.3 Observation

Table 1: Correlation Matrix between variables

Correlations

V1 V2 V3 V4 V5 V6 V7 V8 V9

Pearson -
1 .498(**) -.072 -.330 .152 -.218 -.194 .412(*)
Correlation .478(**)
V1
Sig. (2-tailed) .002 .682 .004 .053 .383 .208 .263 .014
N 35 35 35 35 35 35 35 35 35
Pearson - - .464(**
.498(**) 1 .172 .080 -.196 -.099
Correlation .547(**) .734(**) )
V2
Sig. (2-tailed) .002 .324 .647 .001 .259 .000 .571 .005
N 35 35 35 35 35 35 35 35 35
Pearson -
-.072 .172 1 .709(**) .150 .056 -.119 .268
Correlation .511(**)
V3
Sig. (2-tailed) .682 .324 .000 .389 .751 .002 .494 .119
N 35 35 35 35 35 35 35 35 35
Pearson -
.080 .709(**) 1 .206 -.031 -.293 .099 .157
Correlation .478(**)
V4
Sig. (2-tailed) .004 .647 .000 .234 .860 .087 .570 .369
N 35 35 35 35 35 35 35 35 35
Pearson -
-.330 .150 .206 1 .181 .210 -.237 -.361(*)
Correlation .547(**)
V5
Sig. (2-tailed) .053 .001 .389 .234 .298 .226 .170 .033
N 35 35 35 35 35 35 35 35 35
Pearson -
.152 -.196 .056 -.031 .181 1 .459(**) .084
V6 Correlation .484(**)
Sig. (2-tailed) .383 .259 .751 .860 .298 .006 .003 .631
24 (IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 9, September 2010

N 35 35 35 35 35 35 35 35 35
Pearson - -
-.218 -.293 .210 .459(**) 1 .201 -.343(*)
Correlation .734(**) .511(**)
V7
Sig. (2-tailed) .208 .000 .002 .087 .226 .006 .247 .043
N 35 35 35 35 35 35 35 35 35
Pearson -
-.194 -.099 -.119 .099 -.237 .201 1 -.211
Correlation .484(**)
V8
Sig. (2-tailed) .263 .571 .494 .570 .170 .003 .247 .224
N 35 35 35 35 35 35 35 35 35
Pearson
.412(*) .464(**) .268 .157 -.361(*) .084 -.343(*) -.211 1
Correlation
V9
Sig. (2-tailed) .014 .005 .119 .369 .033 .631 .043 .224
N 35 35 35 35 35 35 35 35 35
** Correlation is significant at the 0.01 level (2-tailed). * Correlation is significant at the 0.05 level (2-tailed).

2.4 Analysis and Discussion 3. Study 2: To compare the performance of

Table 7 is a 9 X 9 symmetric correlation table. Correlation
between all the 9 variables of the LORI chart taken as V1 to
V9 respectively has been shown here. ** marked values
show the Correlation between two variables is significant at
the level of 1% and * indicates the level of significance at
5%. The Correlation table shows that there is a significant
positive correlation between V1 and V2 (r= .498, p<.002)
and V1 and V9 (r=.412, p<.014) however, there is a
negative correlation between V1 and V4 (r=-.478, p<.004).
However, the correlation of V1 with other variables was
found to be non-significant. Similarly, there is a significant
positive correlation between V2 and V9 (r=.464, p<.005)
and negative co-relation between V2 and V5 (r=-
.547,p<.001) and the correlation of the rest of the variables
with V2 are non-significant. A positive correlation exists
between V3 and V4 (r=.709, p<.0001), V6 and V7
(r=.459,p<.006) and a negative correlation between V3 and
V7(r=-.511,p<.002) and V6 and V8 (r= -.484,p< .003). So
the Null hypothesis that there is no correlation between two
variables is rejected wherever the correlation coefficient is
significant.
The above table also suggests that content quality has
positive correlation with learning goal alignment and
motivation which clearly suggests that when the content
quality is enhanced the students are better aligned to their
learning goals and their motivation enhances. The positive
correlation has also been observed between other variables
like Feedback adaptation and Motivation and Interaction
Usability and Accessibility as shown in the table 7. This
indicates that better the design features, better is the
usability and accessibility i.e. they are directly proportional
to one another. So wherever the correlation is significant,
the Null hypothesis that the correlation between the two
variables of LORI sheets is zero, is rejected.
students when taught C++ under traditional,
teacher led instruction mode with the
performance when taught C++ using Learning
Object Approach.
3.1 Aim of the study
The aim of this study is to gain empirical evidence on the
educational effectiveness and usefulness of Learning Objects
in various instructional settings. In this study the
effectiveness of Learning Object learning environment is
compared with traditional classroom teaching environment
in delivering course on C++. So this study becomes an
exploratory study that consists of comparing performance of
two groups of students studying Programming in C++
under two different conditions
• Using traditional teacher led approach
• Using Learning Objects
3.2 Methodology followed
The study was based on pre-test-post-test design in which
the educational effectiveness of Learning conditions
(Learning Object versus traditional instruction) is measured
in terms of gains in subject knowledge. In addition to
statistical analysis of test score data, more descriptive and
qualitative observational analysis were made. The results of
this study were compared and some conclusions were
drawn.
The study consisted of two hours of concrete working and
administration of various tests. Before the actual working
phase (treatment phase), students completed a pre-test to
gain insight into their background about the programming
language C++.
 (IJCNS) International Journal of Computer and Network Security, 25
Vol. 2, No. 9, September 2010

time to explore the topic. At the end of the course both the
Over a period of a semester the students were taught the groups were tested.
course in different modes. The total group size was 35
where 18 students were imparted training using Learning
Object approach and the rest 17 using traditional mode.
Ahead of this group formation the students with no 3.6 Hypothesis
programming experience and previous programming
experience were identified and then the groups were formed Traditional Approach
with both types randomly distributed within the two groups. 1) HO = The teaching using traditional method is not
effective.
3.3 Structure of Teaching Session 2) H1 = The teaching using traditional method is
effective
Each group was given an hour’s session with the following Learning Object Approach
structure 1) HO = The teaching using Learning Object method is
1. A lecture about the C++ topic. not effective.
2. Individual study and use of corresponding learning 2) H1 = The teaching using Learning Object method
materials is effective
3. Solution of a test
Where HO is the Null Hypothesis which is an important
In order to have a controlled teaching style variable the technique used by the researchers in the field of education.
same teacher conducted both the sessions. In the LO A Null hypothesis is useful in testing the significance of
condition (n=18) the students were given introduction on difference.
the subject content to the students first and for the rest of the
sessions the students completed LO assignments 3.7 Analysis and Observations
individually at their own pace. The Learning Objects were
principally quite simple drill-and-practice programs which In order to note and compare the behavioral patterns of the
were designed to be game-like and to provide instant students studying the same course in different modes, the
feedback for students’ input/answers. The way of working teacher conducting the test provided the qualitative data to
was student-led because there was no direct teaching nor the analysis. They asserted the following observations that
teacher controlled tasks during the assignment phase. The during the course of evaluation it was found that the
students were briefed that in order to be successful in this participants of the learning object group were nervous,
approach they required a higher level of self-regulation and stressed and anxious. They took more time to complete the
meta cognitive skills like self-monitoring, controlling, test whereas the traditional group was more at ease, relaxed
maintenance of task orientation etc. than working in the and less stressed. The data obtained for both groups has
traditional condition. been summarized as below:-
The students (n=17) in the traditional mode were taught in
the normal classroom. The teaching method resembled Table 2: Pre-test Post-test analysis of Traditional and LO
normal instruction with a teacher-led introduction followed group
by an assignment phase when students individually Group N Mean Std. Paired p-
completed different paper-and-pencil tasks. These tasks
were similar to the assignments completed in the LO mode. Deviatio t-test value
n
3.4 Description of the Test
Traditiona
The students were evaluated on the basis of a written exam. 17 2.29 2.51 .483 >.05
l Pre-test
The test consisted of two exercises : one to demonstrate
theoretical knowledge of the students and second to judge
their practical programming abilities. The complete test 17 3.11 2.12
Post-test
summed up to a total maximum score of 10 points. The test
was conducted for the one hour duration. LO
Pre-test 18 3.00 2.26 .388 >.05
3.5 Teaching Methodology:
18 3.61 2.10
The traditional group was imparted training using lecture
Post-test
delivery and instructional material containing theory and
examples. Later they were given free time to explore, study
and experiment with the topics covered in the class. The results clearly indicate that the mean of the Learning
However, the lectures were delivered to the Learning Object Object group was slightly superior to the traditional group.
group by using the Learning Objects designed by the They performed better for both the tests given to them as
researcher as part of this study. They too were given free part of the evaluation. In the pre-test the traditional group
(mean=2.29), though, did not perform better than the LO
26 (IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 9, September 2010

group(mean=3.00). However, the performance of the Technology and Distance Learning, 4(3), pp 31-44,
Learning Object group (mean=3.61) was better than the 2007
traditional group(mean=3.11). When a paired t-test was [2] R. Kay & L. Knaack, “Developing learning
applied on the groups to pre and post observations, the t test objects for secondary school students: A
results were non-significant in each case. (Pre-test =.483, multicomponent model”. Interdisciplinary Journal of
Post test =.388). The paired t-test, thus, showed no Knowledge and Learning Objects,1, pp 229-254 , 2005
statistically significant difference between the [3] R. Kay & L. Knaack, “Evaluating the learning in
condition(p>.05). The Null Hypothesis, therefore, learning objects”. Open Learning: The Journal of Open
has to be accepted that there is no significant difference and Distance Learning, 22(1), pp 5–28, 2007.
between the performance of students in both/each modes. [4] S. Nurmi & T. Jaakkola, “Problems underlying the
According to the preliminary analysis , the observation learning object approach”. International Journal of
revealed that in the LO condition students worked mainly on Instructional Technology and Distance Learning,
the perception that they had two different assignments: learn 2(11), pp 61–66, 2005.
to use the Learning Objects and complete the test. They [5] K.Salas & L. Ellis, “The development and
seemed to focus on the procedural features and concrete implementation of learning objects in a higher
functions rather than on content or instructional aspects. education setting”. Interdisciplinary Journal of
They were more interested in solving the logic behind the Knowledge and Learning Objects, 2, pp 1-22, 2006.
Learning Objects i.e. how they worked. Although student’s [6] Felix g,Hamza-Lup, Razvan Stefan Bot, Ioan Salomie.
task orientation remained well during the sessions while “Virtual University of Cluj-Napoca, A Web based
they were actively accomplishing LO assignments, the depth Educational Framework”. [Accessed: Aug 20, 2010]
of orientation was not much. The working with LO did not [7] T. Cochrane, “Interactive QuickTime: Developing
engage students with thinking about the content being and evaluating multimedia learning objects to enhance
learnt. Instead, the work in the traditional condition was both face-to-face and distance e-learning
much more focused on the learning tasks. This could be due environments”. Interdisciplinary Journal of Knowledge
to a larger amount of external control imposed by the and Learning Objects. 2005
teacher. It can, therefore, be concluded that though the [8] D.A. Wiley, “Connecting learning objects to
requirements of self-regulation in the LO condition were instructional design theory: A definition, a metaphor,
overwhelming at the same time they were detrimental to and a taxonomy. In D. A.Wiley (Ed.), The instructional
their learning outcomes. use
of learning objects”: Online version. [Accessed: Aug
It can also be conveniently assumed that the sole use of 20,
Learning Object in lecture impartation and course delivery 2010]
cannot bring a significant difference in the performance of [9] www.nottingham.ac.uk/xerte/.
students. There are other important contextual factors that [Accessed: Aug 20, 2010]
may yet have to be identified to improve academic [10] J.C. Nesbit & J.Li, “ Web-based tools for learning
achievement. Besides, a more prolonged exposure to LO has object evaluation”. Proceedings of the
to be explored and measured under carefully controlled International Conference on Education and
conditions. Also it is imperative to think if the learning Information Systems: Technologies and Applications,
orientation in the LO condition of the students is mere Orlando, Florida, USA, 2, pp 334-339, 2004.
curiosity to a new style of learning for students or the LO
design then supporting pedagogy needs to be modified.
Therefore, when examining the effectiveness of Learning
Objects on student learning outcomes, it is essential to note
that it is the effect of the whole learning environment and
not just of that of Leaning Objects. As it is impossible to
separate learning activities, learned contents and learning
situations from each other, it is also not feasible to detach
the educational technology applications used from the social
and contextual factors of the learning processes. Thus, it is
the Learning Objects and the instructional arrangements
within learning environments that interact together to
stimulate certain student learning activities, behaviours and
outcomes. Learning Object represent only one part of the
larger learning environment and not as self-contained
instructional solution.

References

[1] Y. Akpinar & H. Simsek, “Should K-12 teachers

develop learning objects? Evidence from the field with
K-12 students”. International Journal of Instructional