Interacting with Computers 11 (1999) 453–462

Learning through virtual reality: a preliminary

investigation
Stella Mills a,*, Maria Madalena T de Araújo b
a
Department of IT, Cheltenham & Gloucester CHE, The Park, PO Box 220, Cheltenham GL50 2QF, UK
b
Department de Sistemas e Producção, Universidade do Minho, Azurém, 4800 Guimarães, Portugal
Received 1 December 1997; received in revised form 8 June 1998; accepted 8 June 1998

Abstract
Our understanding of learning through the use of Virtual Reality (VR) is still in its infancy but a
small core of work is emerging that is of growing importance. The literature is utilised to derive three
design principles that are pertinent to VR systems used for learning. These principles form the basis
for the design of a small VR world which was used for teaching a managment technique to students
in Higher Education (HE). Thus, this project naturally divided into two stages: first, software was
developed for Portuguese HE students to learn the basic concept of apportioning resources subject to
constraints, while Stage 2 comprised a formative experiment to test for differences in the learning of
the technique. The conclusion was that overall the traditionally taught group faired better, but not
statistically significantly better, than the software based group. Issues of enjoyment and learning
were also raised. More studies are needed before any generalities can be drawn. q 1999 Elsevier
Science B.V. All rights reserved.
Keywords: Virtual Reality; VR Systems Design; Learning through VR

1. Introduction

Virtual Reality (VR) has become a popular medium for a number of application areas
such as modelling safety critical systems and simulating engineering systems such as
automotive engineering. Another area of application is that of education and VR has
been used successfully in simulating situations for training such as on the flightdeck of
aeroplanes [1]. However, there were few, if any, projects which have attempted to evaluate
the learning achieved through using VR, particularly comparing this with traditional class
learning, perhaps because of the difficulties in designing a tight experiment for such a
comparison. Confounding variables can easily become significant, but even so, compara-
tive methods can be used to expose issues arising from using software for learning. This

* Corresponding author.

0953-5438/99/$ - see front matter q 1999 Elsevier Science B.V. All rights reserved.
PII: S0953-543 8(98)00061-7
454 S. Mills et al. / Interacting with Computers 11 (1999) 453–462

paper describes a pilot study that used a formative method to compare the learning gained
by using VR with that of traditional teaching by a lecturer; assessment was by a short
written test.

2. VR in Education

The terms Virtual Reality and Virtual Environments have come to have a number of
different meanings [2], from what is more exactly described as telepresence to totally
immersive systems in which the user is isolated from the real world. The sense in which
VR is used here is somewhere between these two extremes in that it is non-immersive but
allows the user to move around the computer generated world in the same way as the user
would move around a real world — a true example of non-immersive VR or desktop VR
[3]. Such systems have been used in education for a while; for example, [4] related a fire-
training scenario used by the military, but less safety-critical situations have also been
explored within a learning environment such as laboratories involving scientific experi-
ments and wind-tunnels [5].
All these applications (whether immersive or non-immersive) exploit the visual strength
of VR which is known to be important in gaining conceptual insights [6]. Thus, educa-
tionally, VR is important, and possibly at its best, when visualising situations which are
not generally available to the student; this is why it has yielded apparent success in training
in safety critical situations but it is just as valid in other situations such as planetary
exploration [7] or fashion design [8].
Another area of application is in children’s education. Indeed, it is admitted that in the
US, children do not experience enough learning of skills and concepts which are related to
society’s needs [9] and perhaps VR is a way of addressing this lack, at least in part. It
certainly seems that, theoretically at least, VR systems should be used to simulate a
concept which is difficult for the user to visualise. This could be through lack of experience
or abstraction which can be supplemented for the user by exploiting VR’s strength of
visualisation.
Moreover, it is equally important that the system develops the student’s independent
thinking as well as increasing the extent to which a student learns and retains specific
knowledge [9]. It is also desirable that the system increases the student’s motivation for
learning. Indeed, [10] emphasised the apparent experiential quality of learning through
VR, asserting the the virtual world should be a place in which the student can experience a
sense of presence and interact with the other characters (whatever they may be) within that
place (virtual world). Thus the students are encouraged to use the VR system to help build
their own knowledge worlds which become vehicles of learning [9,10]. In essence, this fits
the psychological theory of constructivism [11] which takes an exploratory approach to
learning. Constructivism has become more popular since the late 1960s [9] and computer
learning generally, and VR in particular, lend themselves to such an approach.
As with any learning, motivation is important and constructivists claim that an explora-
tory method of learning can, at least, maintain motivation and may increase it. However,
more studies are needed in order to learn more about the causes of motivation in students
notwithstanding that motivated students learn more easily than non-motivated students.
 S. Mills et al. / Interacting with Computers 11 (1999) 453–462 455

Thus, if the software can increase motivation then this will enhance the learning of the
students.
Summarising these points, it is important the strengths of VR are used in learning and,
consequently, the system:
1. should exploit the visualisation of the world it portrays;
2. encourage the student to explore and to learn through constructing their own knowledge
patterns;
3. should increase the student’s motivation for learning by allowing the student to feel a
sense of presence within the world.
Of course, these criteria are ideal and it may be that technological advances are needed
before they will become a norm, but they should be the intended goal of all designers of
VR systems for use in education.
The study described here endeavoured to produce a VR world which satisfied the
criteria discussed earlier and this was used to teach Portuguese undergraduates a manage-
ment technique. The experience of the students was compared with a similar group, used
as a control group, and taught traditionally by a lecturer. Thus the project naturally divided
into two stages: that of developing the software and that of conducting the experiment.

3. Stage 1 — software development

The area of application for the software was chosen to be the problem of assigning
resources subject to constraints — a problem very common in operational research. The
particular example was taken from a case-study [12] although only the first part of the
study was implemented. The scenario consisted of planning for the expansion of nursing-
home provision subject to building and travel cost constraints in any one of four geogra-
phical areas. Town A, for example, could accommodate one nursing home of 20 patients
from the four towns after which further homes had to be built elsewhere. Some towns had
sufficient resources for two nursing-homes but travel and time costs for the patients also
had to be minimised, thus exhibiting a problem of where to build a nursing home that
satisfied building restrictions and also limited travel for the patients. Thus the problem was
one of non-linear programming with variable constraints.
While this scenario could be adequately represented in multimedia, a more interactive
situation requiring movement in a three-dimensional world was deemed better as this
would exhibit more obviously the movement of the patients and the restriction of building
in certain towns. However, the drawback to this is the time taken to build the VR world in
the software, which could be two years or more for a fairly simple world such as the
evacuation of a house on fire [4]. Therefore, the chosen scenario was implemented in
software which could be developed in a few months and so within the timescale of the
project. However, while the textures of the software and some of its design could benefit
from better development, it was deemed natural enough by the users at the prototyping
stage for sound understanding by the students. Thus, the software was designed to
exploit the visualisation aspects of VR while encouraging the students to explore and
learn by using buildings as the nursing-homes and three-dimensional movement for the
456 S. Mills et al. / Interacting with Computers 11 (1999) 453–462

Fig. 1. A typical screen.

transportation of the patients. It was intended that the patients would be represented as
people and roads used as links between the towns but unfortunately, software limitations
did not allow this.
The design followed the established method of storyboarding [13] and prototyping. The
initial screen designs had to be modified in order to accommodate the technical limitations
of the software thus illustrating the development still needed in VR development shells in
order to allow the easiest learning environment for the end-user [14]. The major problem

Fig. 2. A typical screen showing mathematical explanation.

 S. Mills et al. / Interacting with Computers 11 (1999) 453–462 457

Table 1
raw results data from test

User Group 1— (Lecture taught) Group 2 — (VR taught)

1 0.25 0
2 0.75 0
3 0.375 0.625
4 0.5 0.375
5 0.375 0.25
6 0.875 0.625
7 1 0.5
8 0.375 0.875
9 0.625 0.625
10 1.125 0
11 0
Average (Mean) 0.525 0.352

here was ensuring that the user could see the method of calculation used to arrive at the
result; this was important as the underlying theory embedded in the calculation was
essential for a complete understanding of the management method and without this theory
further application of the method would be impossible. Eventually it was decided to give
the mathematical explanation of the technique in a dialogue box at the end of each short
section of the implementation (Fig. 2). However, reading and understanding these could
easily be ignored by weaker students, particularly those who completed the sections
correctly. Consistency was maintained between screens by using similar designs for
each screen and each overall section. Figs. 1 and 2 show examples of screens used in
the system.
On completion, the software was evaluated as a prototype by a small group of users
(students and two members of the teaching staff) at the Portuguese University and conse-
quently, some small changes were made to the language (Portuguese) and to the third
(final) section of the software in order to achieve better consistency. It should be noted (see
Fig. 2) that the software did not support accents etc. being added to the text. These users

Fig. 3. SUMI scores.

458 S. Mills et al. / Interacting with Computers 11 (1999) 453–462

also made a (subjective) evaluation of the software using the criteria discussed earlier as
heuristics and all the users declared the software satisfied them on all accounts.
In addition, the software was evaluated after use by asking each user to complete a
SUMI questionnaire [15] which yielded ratings (see Table 1) for learnability, efficiency,
affect, helpfulness and control as well as giving an overall usability rating. Full discussion
of these ratings can be found elsewhere [14]; suffice to say that a rating of 50 indicates
software which is ‘state of the art’ [15] and the software here rated overall at 47. From Fig.
3, it can be seen that the software had high ratings of learnability (62) and helpfulness (54)
while control and efficiency were low at 41 and 42, respectively. Affect was adequate at
49. However, these divergent results were probably caused by the fact that the instrument
caters for different types of software such as spreadsheets and databases and thus includes
questions about data-entry and data-processing. In a VR World, these types of input do not
usually occur, so the scores for efficiency and control which relate to ease of (data)
manipulation and interaction through user input are skewed accordingly. This raises the
need for a more VR specific type of instrument to be developed.

4. Stage 2 — evaluation of learning

The evaluation of learning has evolved as a global term with a variety of different
methods of assessment. However, in this case, it was decided that since the project was
using real undergraduate material, the learning should be assessed in the traditional way of
an examination. While it is appreciated that this may not be the best method of assessment,
it is the method normally used within the Portuguese university, and therefore was realistic
within the experimental environment. Consequently, a short written test, heavily focusing
on the material learnt, was given. At this preliminary stage of experimentation, no allow-
ance was made for confounding variables such as different performances under examina-
tion conditions. It is worth repeating that designing a summative experiment in order to
evaluate learning is fraught with difficulties caused by confounding variables and, conse-
quently, a formative approach has been used in order to identify issues for further work.

4.1. Method

In order to evaluate the learning experience through VR, the experimenters selected a
case-study [12] which was used as the instrument for the software and was also taught to
students by a lecturer, who had a large amount of experience in teaching this work and had
also received good teaching testimonials from past students. An after-only experimental
research design of independent subjects [16] was chosen, with a group of students being
taught traditionally by a lecturer as the control group and a second equally sized group of
similar students using the software as the experimental group. From the prototyping
evaluation, the general learning time seemed to be around 10 minutes, so the experimental
group was allowed to use the software for a maximum of 30 minutes in order to adequately
learn the resourcing technique while the control group was taught in a class in the usual
way. All the students then were asked to complete a single question test which utilised the
technique learnt, and the results were compared. This was taken to be synonymous with
 S. Mills et al. / Interacting with Computers 11 (1999) 453–462 459

Fig. 4. Results of Group 1 (traditionally taught students).

the ability to apply the knowledge learnt to a similar scenario to that used in the learning
experience.
The question for the test was based very closely on the technique learnt and required
only a change of scenario; in place of a healthcare environment an organ pipe manufac-
turer was chosen. Corresponding data were changed as well but the question intentionally
related very closely to that of the original case-study.

4.2. Participants

Twenty-one students from a Department of Engineering Production from a Portuguese

university took part in the experiment. They were all final year undergraduate students of
Engineering and had volunteered to take part. The total group was representative of the
final year cohort in terms of academic excellence, age and gender. Although none of the
students had studied material similar to that used in this project, they all had some
mathematical knowledge as well as computer systems engineering.

Fig. 5. Results of Group 2 (VR taught students).

460 S. Mills et al. / Interacting with Computers 11 (1999) 453–462

4.3. Procedure

The students were divided voluntarily into two groups of 10 and 11 students each, the
smaller group (Group 1) being taught by the lecturer and the larger group (Group 2) using
the software. After being briefed that they would be given a maximum of 30 minutes to use
the software, the larger group was allowed to study the technique through the software
repeating as many times as they wanted. When each student indicated that they had learnt
the management technique from the software (which was well within the scheduled 30
minutes, they were asked to complete a copy of the SUMI questionnaire [15]. At the same
time but in an adjacent room, the smaller group of students were taught the technique by
the lecturer. After each learning session had finished, the students were given a 10 minutes
break. On return, they were all shown into the examination room and given the test to be
completed under normal examination conditions.

4.4. Results

The raw results of the experiment are given in Table 1. In general, the results for the
software students (Group 2) were disappointing on two accounts: first, only nine of the 11
students returned from the break to take the test and secondly, two of these failed to score
so that only seven students gained positive marks (Fig. 4). The 10 students who learnt with
the lecturer traditionally (Group 1) all scored, one student (Number 7) actually gaining full
marks (Fig. 5). The number of participants is too small to be subject to a full statistical
analysis, but the Mann–Whitney Test revealed that while the traditionally taught group
(Group 1) performed better than the software taught group (Group 2), the results are not
statistically significantly better for 0.005 error tolerance. Consequently, the null hypoth-
esis is proved.
However, this does not address the issue of why students did not return from the break
and why, of those who did return, two did not score at all. However, since anonynmity
prevailed and no attempt was made to record other factors of the students such as indivi-
dual learning preferences, no further conclusions about this can be drawn from this data
and the total number of scores of Group 2 must be used for data analysis.

4.5. Discussion

The most important issue arising from the results is that the difference between the
means of the two groups suggests that the traditionally taught students learnt more thor-
oughly and were more able to apply their knowledge than the computer taught students.
However, the samples were far too small to generalise this [17,18] and, in any case,
explanations should first be sought from the data themselves. We should note that the
Mann–Whitney Test revealed there was no significant difference between the two sets of
results for an error tolerance of 0.005 which was supported by similar results from t-tests.
Unfortunately, a dispersion analysis was not possible using the Mann–Whitney Test since
the means of the two groups are different. Even though there is no statistical evidence to
suggest any quantative difference between the two groups, behavioural evidence indicates
that more qualitative data may have revealed some differences.
In the first place, students were given the choice of learning traditionally or by software
 S. Mills et al. / Interacting with Computers 11 (1999) 453–462 461

but the experimental design did not allow for investigating the learning preferences of the
students or their psychological types. Thus it may be that the types of students who are
more easily suited to certain learning styles were not amicably matched with the learning
environments. The software was designed to exploit the constructivist approach to learn-
ing and it could be that some students in Group 2 preferred to learn in a more instructional
way. This could explain the negative reaction of the two students who failed to take the test
as well as those who scored less ably using the software. Further work is now needed to
correlate the learning preferences, the psychological types and the SUMI results of the
students.
Another problem was that the case-study used an instructional approach by the nature of
the problem in that all mathematical exercises are sequential. VR emphasises the visual
aspect of learning particularly in an exploratory way which may not be the best method for
learning sequential tasks. In addition, the software prevented the interface from having a
naturally textured landscape that added cognitive load for the user [19]. The structure of
the system intentionally did not emphasise the mathematical method, thus requiring the
student to think ahead in terms of future applications of the method. The task of calculating
the distances in the problem was assigned to the computer since this is a mundane task, but
this meant that a part of the method was removed from the student’s sequential learning to
a dialogue box only visible at the end of the period. Consequently, a student in Group 2 did
not see all aspects of the complete method receiving the same emphasis — as, in fact, the
traditionally taught students did since the lecturer taught the actual calculation as part of
the method. Again, the visual and constructivist approach which seems so excellent for VR
applications may have caused the student to ‘gloss over’ the very details that s/he would
need for the test.
Turning now to the more general concept of the learning experience using VR, [10]
described a case-study which involved students at summer school using VR to create their
own worlds based on their experiences of life. The school lasted a week and the students
were asked if they had enjoyed their time and whether they wanted to experience VR
again. ‘Enjoyment’ scored 6.5 and ‘repeating the experience’ scored 6.8 on a scale which
ran from 1 to 7, but when asked if they felt that VR was a good learning environment, the
students rated this at 5.7. The authors do not suggest reasons for the difference in these
scores but it suggests there is need for further investigation. While these results do not
explain the failure of the two students to return after the break, these experimental findings
do echo those of [10] in that both sets of results seem to indicate a difference in concept for
the users between enjoying using a VR system and actually learning from it. Clearly,
further work is required to investigate possible reasons for this.

5. Conclusion

The project has shown that there may be a difference in learning in higher education
through virtual reality when compared with learning traditionally. However, if this is so
then further work needs to be done to investigate the causes; in particular, the design of the
software and student learning preferences, including learning styles and psychological
types, need to be analysed more closely. Indeed, more studies of performance and learning
462 S. Mills et al. / Interacting with Computers 11 (1999) 453–462

preferences are needed in order to ascertain whether there is correlation between these two
in the case of computer supported learning. Of course, this requires the sound design of
studies that has been shown here to be fraught with confounding variables. However,
summative studies could give useful information about the value of learning with VR.
Further trends concerning enjoyment and learning need to be investigated too. Finally,
there is a need for an instrument that can be used to evaluate the usability of virtual
environments so that the software’s worth per se can be ascertained. These advances
would help greatly in the evaluation of using virtual environments for learning.

Acknowledgements

The authors wish to thank The British Council for their financial support in this project.
They also appreciate the kindness of Superscape plc. in allowing the free use of a full
evaluation copy of vrt 4.00 for the duration of the project.

References

[1] L. MacDonald, J. Vince, Interacting with Virtual Environments, Wiley, Chichester, 1994.
[2] S.R. Ellis, Pictorial Communication in Virtual and Real Environments, 2nd ed., Taylor and Francis,
London, 1993.
[3] S. Mills, M.M.T. de Araújo, Distance Learning by Virtual Reality — Two Preliminary Issues, Video and
Computer Conferencing in Education and Training Conference, 23–24th September 1996, University of
Aberdeen, Inverness.
[4] C. Mason, Desktop Virtual Reality in the RAF, Paper given at Multimedia in Business Conference, June
1996, Cheltenham and Gloucester College of Higher Education.
[5] R.S. Kalawsky, The Science of Virtual Reality and Virtual Environments, Addison–Wesley, Wokingham, 1994.
[6] M. Kaiser, Knowing, in: S.R. Ellis, Pictorial Communication in Virtual and Real Environments, 2nd ed.,
Taylor and Francis, London, 1993.
[7] M.W. McGreevy, Virtual Reality and Planetary Exploration, in: A. Wexelblat (Ed.), Virtual Reality Appli-
cations and Explorations, Academic Press, London, 1993.
[8] N.M. Thalman, Tailoring Clothes for Virtual Actors, Chapter 13 in MacDonald and Vince, 1994.
[9] N.I. Durlach, A.S. Mavor (Eds), Virtual Reality Scientific and Technological Challenges, National
Academic Press, Washington, DC, 1995.
[10] M. Bricken, C.M. Byrne, Summer Students in Virtual Reality, in: A. Wexelblat (Ed.), Virtual Reality
Applications and Explorations, Academic Press, London, 1993.
[11] P. Fensham, R. Gunstone, R. White, The Content of Science — A Constructivist Approach to its Teaching
and Learning, The Falmer Press, London, 1994.
[12] S.R. Heinmann, E.J. Lusk, Health facility planning: an example of a decision flexibility approach, Opera-
tional Quarterly Review, 27 (2, ii) (1979) 449–457.
[13] M.J. Bunzel, S.K. Morris, Multimedia Applications Development using IndeoVideo and DVITechnology,
McGraw-Hill, New York, 1994.
[14] S. Mills, M.M.T. de Araújo, Learning from virtual reality — a lesson for a designer, HCI Letters, 1 (1998)
28–31.
[15] SUMI, Software Usability Measurement Inventory, University of Cork, Ireland, 1996.
[16] L.B. Christensen, Experimental Methodology, 4th ed., Alleyn and Bacon, Boston, MA, 1988.
[17] P. Johnson, Human Computer Interaction Psychology Task Analysis Software Engineering, McGraw-Hill,
Maidenhead, 1992.
[18] G. Lingaard, Usability Testing and Evaluation, Chapman and Hall, London, 1994.
[19] K. Carr, R. England, Simulated and Virtual Realities Elements of Perception, Taylor and Francis, London,
1995.