A comparison of two-dimensional prediction tracing and a virtual reality patient methods for

diagnosis and treatment planning of orthognathic cases in dental students: a randomized

preliminary study.
Authors: Sakowitz, Scott M.1 (AUTHOR)
Inglehart, Marita R.2 (AUTHOR)
Ramaswamy, Vidya3 (AUTHOR)
Edwards, Sean4 (AUTHOR)
Shoukri, Brandon1 (AUTHOR)
Sachs, Stephen5 (AUTHOR)
Kim-Berman, Hera1 (AUTHOR) bermanh@umich.edu

Source: Virtual Reality. Sep2020, Vol. 24 Issue 3, p399-409. 11p.

Document Type: Article

Subject Terms: Dental students

Simulated patients
Forecasting
Virtual reality
Diagnosis methods
Dental clinics
Concept mapping
Two-dimensional models

Author-Supplied Dental education

Keywords: Oral surgery
Orthodontics
Orthognathic surgical prediction
Simulated patient

Abstract: Virtual reality patient (VR patient), a simulated patient module in a virtual reality environment allowing manipulation of the upper and lower jaws and chin in three planes of space, was developed to help students
understand diagnosis and treatment planning of orthognathic surgical procedures. The objective was to compare student understanding in diagnosing and treatment planning complex orthognathic cases using the
VR patient versus a conventional 2D prediction tracing method and to determine feasibility of utilizing VR methods. Thirty third year dental students were assigned randomly to an experimental (VR patient) or control
(2D tracing) group. The dependent variables were a multiple choice question (MCQ) examination, baseline and exit surveys, and written case analysis of two cases. Student–teacher interactions were recorded for
both length and type of interaction. Data were evaluated using descriptive and inferential statistics. The students' performance on the MCQ examinations improved immediately following the educational intervention
(p <.05). However, no significant difference was found between the 2 groups on the written case analysis and pre-test, post-test, and follow-up MCQ examinations. The effect size of the intervention ranged from.14
to.90 and differed greatly between the written responses to the two cases. Intra- and inter-rater reliability of the written response scoring was found to be reliable and reproducible (>.928). Dental students were able
to improve their understanding of diagnosis and treatment planning of orthognathic cases using both 2D prediction tracing and the VR patient methods. The method of scoring the written responses was reliable and
reproducible and should be used for future full-scale studies. [ABSTRACT FROM AUTHOR]

Copyright of Virtual Reality is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However,
users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material
for the full abstract. (Copyright applies to all Abstracts.)
1
Author Affiliations: Department of Orthodontics and Pediatric Dentistry, School of Dentistry, University of Michigan, 1011 N. University Ave, 48109-1078, Ann Arbor, MI, USA
2Department of Periodontics and Oral Medicine, School of Dentistry, University of Michigan, Ann Arbor, MI, USA
3School of Dentistry, University of Michigan, Ann Arbor, MI, USA

4
Department of Oral and Maxillofacial Surgery, School of Dentistry, University of Michigan, Ann Arbor, MI, USA
5New York Center for Orthognathic and Maxillofacial Surgery, Lake Success, NY, USA

Full Text Word 7579

Count:

ISSN: 1359-4338

DOI: 10.1007/s10055-019-00413-w

Accession Number: 145302335

A comparison of two-dimensional prediction tracing and a virtual reality patient methods for diagnosis and
treatment planning of orthognathic cases in dental students: a randomized preliminary study
This content may contain URLs/links that would redirect you to a non-EBSCO site. EBSCO does not endorse the accuracy or accessibility of these sites, nor of the content therein. ✖

  Listen
 

Virtual reality patient (VR patient), a simulated patient module in a virtual reality environment allowing manipulation of the upper and lower jaws and chin in three planes of space, was developed to help students
understand diagnosis and treatment planning of orthognathic surgical procedures. The objective was to compare student understanding in diagnosing and treatment planning complex orthognathic cases using the VR
patient versus a conventional 2D prediction tracing method and to determine feasibility of utilizing VR methods. Thirty third year dental students were assigned randomly to an experimental (VR patient) or control (2D
tracing) group. The dependent variables were a multiple choice question (MCQ) examination, baseline and exit surveys, and written case analysis of two cases. Student–teacher interactions were recorded for both
length and type of interaction. Data were evaluated using descriptive and inferential statistics. The students' performance on the MCQ examinations improved immediately following the educational intervention (p <.05).
However, no significant difference was found between the 2 groups on the written case analysis and pre-test, post-test, and follow-up MCQ examinations. The effect size of the intervention ranged from.14 to.90 and
differed greatly between the written responses to the two cases. Intra- and inter-rater reliability of the written response scoring was found to be reliable and reproducible (>.928). Dental students were able to improve their
understanding of diagnosis and treatment planning of orthognathic cases using both 2D prediction tracing and the VR patient methods. The method of scoring the written responses was reliable and reproducible and
should be used for future full-scale studies.

Keywords: Virtual reality; Dental education; Orthognathic surgical prediction; Simulated patient; Orthodontics; Oral surgery

Background
The evolution of modern medical imaging technology and the advent of virtual reality technology has lead to advanced visualization methods to evaluate patient image data for the purposes of educational and clinical
practice (Abhari et al. [ 3]; Agarwal et al. [ 6]; Cao and Cerfolio [12]; Izard et al. [23]; Robb [43]; Uppot et al. [51]; Xin et al. [57]). Since the discovery of the X-ray by Roentgen in 1895 and, beginning in the 1970s, the
development of true digital imaging and dynamic spatial reconstruction capabilities including computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) clinicians have
been able to view and process these medical images from film to high-resolution screens and head-mounted displays in order to interpret and guide diagnosis, intervention, and treatment (Aaker et al. [ 1]; Abdelwahab et
al. [ 2]; Robb [43]; Vertemati et al. [52]; Zawy Alsofy et al. [59]).

From 1989 to the present, applications in medical virtual reality include simulation surgery for tendon transplants, abdominal surgery, and virtual endoscopy among others (Akay and Marsh [ 7]; Abhari et al. [ 3]; Khan et
al. [26]; Wong et al. [55]; Zheng et al. [60]). Other medical applications include interpersonal communication training, exposure therapy, anatomy instruction, diagnosis, and treatment planning. Virtual humans and
characters have been used to research interpersonal scenarios for medical trainees to interact with virtual assistant and virtual patients (Aebersold et al. [ 4], [ 5]; Badler et al. [ 9]; Dyer et al. [16]). Investigators found that
exposure therapy in the virtual environment was as effective as in the real environment for patients with acrophobia, arachnophobia, public speaking, and fear of flying (Carl et al. [13]; Emmelkamp et al. [17]; Minns et al.
[32]; Pertaub et al. [40]; Rothbaum et al. [44]). The use of virtual reality in reducing pain and anxiety in adults and children have also been investigated (Mallari et al. [29]; Pourmand et al. [41]; Sikka et al. [46]; Sweta et
al. [48]; Walther-Larsen et al. [53]; Won et al. [54]). Virtual reality educational tools for human anatomy, dental anatomy, and maxillofacial surgery planning have been developed; however, the available tools are few in
number and the use of virtual reality technology in dentistry is still in its infancy (Arikatla et al. [ 8]; Germans et al. [19]; Dias et al. [15]; Grabowski et al. [20]; Izard et al. [22]; Neumann et al. [34]; Triepels et al. [49]; Xia et
al. [56]; Zaragoza-Siqueiros et al. [58]).

Currently, most graduate orthodontic graduate programs in the USA and worldwide use conventional two-dimensional (2D) methodology when teaching their residents diagnosis and surgical treatment planning. This 2D
method consists of using manually traced lateral cephalometric radiographs where upper and lower jaw and chin can be cut out from the tracing and then moved to help plan the surgery (Proffit and White [42]). Digitized
tracings utilizing computer software can also be used to perform these predictions. These methods have limitations because they only show predictions in the sagittal plane; asymmetry problems of the upper and lower
jaw and subsequent rotational corrections cannot be adequately simulated. Additionally, these conventional 2D methods may seem outdated or inadequate to students who are familiar with new technology and
accustomed to visualizing 3D images (Smith and Foley [47]). Recent advances in three-dimensional (3D) virtual surgical planning (VSP) allow clinicians to plan jaw surgery using proprietary virtual treatment planning
software (Becker et al. [10]). This software combines data from cone beam computed tomography (CBCT) and dental casts and, with the help of a software engineer who performs the virtual simulation, allows the
clinician to treatment plan the surgery and fabricate a 3D-printed surgical splint (Gateno et al. [18]). However, VSP is limited in that it is viewed using a 2D computer monitor and is generally not available to students at
the initial diagnosis and treatment planning stage due to cost and lack of availability of the proprietary software.

Leveraging emerging technologies, a custom virtual reality patient (VR patient) model was developed for orthognathic surgery simulation in this study (UMichDent [50]). The VR patient allows students to explore
diagnosis and treatment planning of orthodontic surgical cases, while placing them in an action-based and interactive learning environment. The VR patient is a simulated patient module created from patient CBCT data.
The upper and lower jaw, and chin of the VR patient can be manipulated in all three planes of space to simulate jaw surgeries such as a bilateral sagittal split osteotomy (BSSO), one- or two-piece LeFort I osteotomy,
and genioplasty. The VR patient is not meant to be an accurate representation of a specific expected surgical outcome, but rather a teaching tool for students to help provide an understanding of the surgery and
diagnosis of problems. The student views the VR patient using a virtual reality head-mounted device and interacts with the VR patient to diagnose and simulate jaw surgical procedures. Students can use the VR patient
to try multiple surgical procedures and movements with relative ease using a virtual reality head-mounted device.

Many virtual reality head-mounted devices are commercially available, but for the purposes of this study, the Oculus Rift (Oculus VR, Menlo Park, CA) will be discussed in detail. The Oculus Rift virtual reality headset
was introduced in 2012 (Oculus Rift History—How it All Started [38]). The first consumer version was released in 2016, marking the first major commercial release of an affordable virtual reality headset with sensor-
based tracking (Oculus Rift CV1 Teardown [35]). The Oculus Rift uses an accelerometer, gyroscope, magnetometer, and infrared sensors to track orientation of the headset (Oculus Rift Development Kit 2 Teardown [36];
Oculus Rift DK1 Teardown [37]; Oculus Rift History—How it All Started [38]).

Although increases in fidelity of the VR learning tool should enhance the student and clinician's ability to understand, the relationship between fidelity and improved outcomes in education with respect to VR still remains
unclear. When Gutierrez et al. compared knowledge acquisition using fully immersed virtual reality (using a head-mounted display) versus partially immersed virtual reality (using a computer monitor), they found that the
immersed group showed a significantly higher increase in knowledge than the partially immersed group (Gutiérrez et al. [21]). However, Buttussi and Chittaro found that increased immersion did not significantly affect the
increase in knowledge (Buttussi and Chittaro [11]).

The objective of this study was to evaluate the feasibility of the testing methods and compare student understanding in diagnosing and treatment planning complex orthognathic cases using the VR patient versus the 2D
prediction tracing method.

Methods
The Health Sciences and Behavioral Sciences Institutional Review Board (IRB) at the University of Michigan determined that this research was exempt from IRB oversight (#HUM00113004).

The study utilized a two-group randomized pre-test/post-test/follow-up test comparison design (Fig. 1) (McMillan and Schumacher [30]). The dependent variables were a multiple choice question examination (MCQ),
baseline and exit surveys, and a written analysis of two cases. During a 3-month period, thirty (30) third year dental students were recruited for this preliminary study and assigned randomly to an experimental (VR
patient) and a control (2D) group. A study coordinator randomly assigned the students to two groups using envelopes that contained a group number and random 5 digit number for deidentification. The primary
investigator did not have access to the master list of participant names and ID numbers. All participants signed an informed consent form and 2 weeks prior to the educational intervention, participants completed a pre-
test (T1) consisting of 12 multiple choice questions (MCQ) and a baseline survey which contained questions concerning their demographic background, video game experience, preferred methods of learning, interest
and experiences in orthodontics and oral surgery, confidence in diagnosing and treatment planning orthodontic cases, and grades received in orthodontics and oral surgery classes.

Graph: Fig. 1 Flowchart outlining the study design. Group 1 = Experimental group = VR patient group. Group2 = Control group = 2D tracing group

For the educational intervention, both the experimental and control groups viewed the same 30 min pre-recorded lecture on orthognathic diagnosis and surgical treatment planning. After the lecture, participants were
informed that they would be diagnosing and treatment planning three patient cases in a hands-on exercise. Case 1 was a practice case and was not evaluated; Cases 2 and 3 were graded and scores were included in
the data analysis.

The experimental group was provided with verbal instructions and a demonstration on how to interact with and perform a surgical simulation on Case 1 using the VR patient (Fig. 2). The participants utilized mouse and
keyboard controls and an Oculus Rift DK2 virtual reality headset (Oculus VR, Menlo Park, CA) and remained seated during the exercise (Fig. 3) (Chiarovano et al. [14]).

Graph: Fig. 2 Frontal and lateral views of the virtual reality patient model

Graph: Fig. 3 Image of participant wearing the Oculus Rift headset and interacting with the virtual reality patient model using keyboard and mouse (color figure online)

The control group was provided with verbal instructions and a demonstration of how to complete a surgical prediction tracing modified from the template method of Proffit and White. (Proffit and White [42]) Participants
were provided with a posterior–anterior cephalometric radiograph, traced lateral cephalometric radiograph, and labeled transparencies with a traced upper and lower jaw, and chin (Fig. 4).

Graph: Fig. 4 Image of traced lateral cephalometric radiograph and transparency overlays (yellow = tracing overlay of original image, red = tracing of upper and lower jaw overlays which has been manipulated)

The participants were allowed up to 30 min to complete each case. The amount of time each participant spent on each case was recorded. For each case, the participants were provided with a treatment planning
worksheet that was modified from the Board Case Oral Examination (BCOE) worksheet utilized by the American Board of Orthodontics (Orthodontics https://www.americanboardortho.com/media/1164/bcoe-examinee-
worksheet.pdf). Scores of the participants' written case analyses of Case 2 and 3 were based on two categories—recognitions/omissions and correct/incorrect answers. For example, if the correct diagnosis was "convex
profile" and a participant wrote "concave profile," he or she would receive one point in the recognition category for recognizing the profile and receive 0 points for the correct category for incorrectly diagnosing the profile.

The grading key for the written case analysis was completed with the help of seven orthodontic faculty experts. Correct answers and recognition items were determined by majority consensus of the expert's response on
their worksheets and compiled for grading in the areas of diagnosis, treatment objectives, and treatment plan. Intra- and inter-rater reliability testing was completed to determine the reliability of scoring the written case
analysis of the study participants.

During the educational intervention, student–teacher interactions were logged for both length of interactions (time in seconds) and type of interactions (Jasinevicius et al. [24]). The types of interactions were categorized
as technical, surgical prediction, diagnosis, treatment planning, and other.

Once participants completed their hands-on exercise, they responded to a post-test (T2) of the same 12 MCQ's that had been part of the pre-test and to an exit survey. The exit survey contained questions that were
identical to the baseline survey as well as questions about the level of engagement during the hands-on exercise.

Two weeks after the post-test, participants were asked to complete a follow-up test (T3) containing the same 12 MCQ's they had answered in the pre- and the post-test.

Descriptive statistics such as frequency distributions, means, standard deviations, and ranges were calculated to provide an overview of the responses. Two-way repeated measure ANOVA (p <.05) was used to
determine if there was a difference between the groups over the 3 testing times for MCQ examinations, T1—pre-test, T2—post-test, and T3—follow-up test. Post hoc paired t test (p <.05) was used for pairwise
comparisons between the testing time points. Student's t test was used to compare the written responses of the experimental and control groups, and chi-square test was used to evaluate student–teacher interactions in
relation to time and type. Intra- and inter-rater reliability in grading the written responses was tested with intraclass correlation coefficient (ICC). Data were analyzed using SPSS (IBM Corp. Released 2013. IBM SPSS
Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.). The effect size of the intervention was determined for the written responses using nQuery + nTerim (Statistical Solutions Ltd, Version 4.0. Boston, MA).

Results

Sample description and retention

Twenty-eight (28) subjects participated in this study, thirteen (13) in the experimental group and fifteen (15) in the control group. Two subjects in the experimental group dropped out before the actual study took place;
one subject in the experimental group did not complete the exit survey and the follow-up MCQ test. Overall retention rate of both groups was 97%, while retention rate for the experimental group was 87% and the control
group was 100%. The experimental group completed the intervention at the University of Michigan 3D Lab and the control group completed the intervention in the University of Michigan, School of Dentistry. Overall, the
responses to the baseline survey showed that the two groups were not significantly different in their video game experience, in how much they liked to learn from various teaching methods, their interest in orthodontics
and oral surgery, their confidence in diagnosing and treatment planning orthognathic surgery cases, experiences in oral surgery, and their grades in orthodontics and oral surgery classes. Although, there was a
significant difference in whether the participants in the two groups had any experiences with orthodontics prior to dental school (experimental group: N = 5, control group: N = 12, p =.025), the experiences in orthodontics
largely consisted of the subjects having orthodontic treatment as an adolescent.

The exit survey showed that the two groups were not significantly different in their interest in orthodontics and oral surgery, their confidence in diagnosing and treatment planning orthognathic surgery cases, and their
level of engagement and interest in the educational intervention. There were no significant differences in answers to other questions such as how much they liked to learn from various teaching methods.

Multiple choice question test at T1, T2, and T3

The multiple choice questions (MCQs) were case based and students were asked to recognize the most suitable diagnosis and treatment approach from a list of options. The mean scores of the correct responses and
standard deviations of the experimental and control groups at T1 (pre-test), T2 (post-test), and T3 (follow-up test) are shown in Table 1. From T1 (pre-test) to T2 (post-test), the students in both groups showed a
significant increase in their test scores (p <.05; Fig. 5). The experimental group improved from 7.33 to 8.42, and the control group improved from 7.93 to 8.93. Although the mean scores were increased at the follow-up
test at T3 in relation to student's baseline scores at T1 for both groups, there was no statistical difference when T3 (follow-up test) scores were compared with both T1 and T2 (pre- and post-test) scores. Additionally,
there was no significant difference in MCQ test scores between the experimental and control groups at the pre, post, and 2 week follow-up time points.

The mean scores of correct responses and standard deviations of the multiple choice question (MCQ) test for experimental and control groups at T1 (pre-test), T2 (post-test), and T3 (follow-up test)

MCQ test Group Mean scoreSD

Pre-test (T1) Experimental (N = 12)7.33 1.37

Control (N = 15) 7.93 2.12

Post-test (T2) Experimental (N = 12)8.42 1.56

Control (N = 15) 8.93 1.71

Follow-up test (T3)Experimental (N = 12)8.17 1.12

Control (N = 15) 8.13 2.23

Graph: Fig. 5 From pre-test to post-test, the students in both experimental and control groups showed a significant increase in their test scores (p <.05). Although the mean scores were increased at the follow-up test in
relation to student's baseline scores at pre-test for both groups, there was no statistical difference when follow-up test scores were compared with pre- and post-test scores

Written responses to Case 2 and 3 for diagnosis, objectives, and treatment plan
The scores on the written case analysis of the two groups were not significantly different in the recognition scores on Case 2 and Case 3 for diagnosis, objectives, treatment plan, and combined total score (Table 2).
There was also no significant difference in the correct response scores for diagnosis, objectives, treatment plan, and total score for Case 3. However, the scores for correct responses in the diagnosis (p =.026) and total
correct responses score (p =.021) on the written case analysis were significantly different in that the control group performed better than the experimental group for Case 2 (Table 2). There was no significant difference in
the correct response scores for objectives and treatment plan for Case 2.

Scores on written case analysis during hands-on exercise by group, recognition, and correct responses for Case 2 and Case 3 (*p <.05)

Scores on written case analysis: (maximum correct response)Recognition response Correct response

VR group 2D group p valueVR group 2D group p value

Mean (SD) Mean (SD) Mean (SD)Mean (SD)

Range Range Range Range

Case 2—Diagnosis 4.00 (.82) 4.47 (1.13) .227 2.85 (.99) 3.93 (1.39) .026*

(7) 3–5 3–7 1–4 2–7

Case 2—Objectives 2.92 (.64) 3.67 (1.40) .077 2.31 (.86) 3.07 (1.22) .072

(6) 2–4 2–6 1–4 2–6

Scores on written case analysis: (maximum correct response)Recognition response Correct response

VR group 2D group p valueVR group 2D group p value

Mean (SD) Mean (SD) Mean (SD)Mean (SD)

Range Range Range Range

Case 2—Tx plan 2.23 (.73) 2.33 (.72) .712 1.00 (.71) 1.27 (.70) .328

(3) 1–3 1–3 0–2 0–2

Case 2—Total score 9.15 (1.35) 10.53 (2.95).119 6.15 (1.77) 8.27 (2.63) .021*

(16) 7–11 6–17 4–10 4–15

Case 3—Diagnosis 5.77 (1.48) 5.53 (1.81) .711 4.54 1.51) 4.40 (1.60) .816

(9) 3–8 2–8 3–7 2–7

Case 3—Objectives 2.46 (.78) 2.80 (1.15) .377 2.46 (.78) 2.80 (1.15) .377

(5) 1–4 1–5 1–4 1–5

Case 3—Tx plan 2.69 (.86) 2.93 (.59) .403 1.54 (1.20) 2.07 (1.10) .235

(4) 2–4 2–4 0–4 0–4

Case 3—Total score 10.92 (2.10)11.27 (2.96) .730 8.54 (2.60) 9.27 (2.96) .499

(18) 6–13 5–16 5–13 4–13

Intra-rater reliability of the written response scoring was determined using intraclass coefficient correlation (ICC) performed at 95% confidence intervals based on a single rater (k = 1), absolute agreement and 2-way
mixed-effects model. Inter-rater reliability was determined using ICC estimates at 95% confidence intervals based on a mean-rating (k = 2), consistency and 2-way random-effects model. Intra-rater reliability ICC values
ranged from.93 to 1.00. Inter-rater reliability ICC values ranged from.96 to 1.00. ICC values greater than.90 indicate excellent reliability. Time spent on each case analysis is shown in Table 3. The experimental group
spent less time completing the treatment planning worksheet for Case 2 (experimental group: mean = 15 min, 5 s, control group: mean = 19 min, 33 s; p =.047; Table 3. However, there was no significant difference in
how much time they spent on Case 3.

Time spent on each case in seconds (*p <.05)

Experimental groupControl groupp

Mean (SD) Mean (SD)

Range Range

Case 2903 (343) 1160 (308) .047*

431–1589 670–1648

Case 3935 (217) 913 (243) .911

590–1311 560–1340

For the student–teacher interactions, the two groups were significantly different in the total number of interactions (experimental group: mean = 3.08, control group: mean =.67, p =.005) and total time of all interactions in
seconds (experimental group: mean = 81.38, control group: mean = 12.07, p =.006). However, when the interactions related to questions about the virtual reality technology were omitted in the analysis, there was no
significant difference between the groups in the number of interactions and the length of time of the interactions for the surgical prediction, diagnosis, treatment planning, and other categories.

The effect size of the intervention, to determine the magnitude of the difference between the two groups, was calculated at p =.05 for total score for recognition and total score for correct responses for Case 2 and Case
3 (Table 4). Estimated effect size using an acceptable statistical power (.80) was variable between the cases, with range from.14 for Case 3 to.90 for Case 2.

Estimated effect size for total score for recognition and total score for correct written responses for Case 2 and Case 3 (p =.05)

Case 2 Case 2 Case 3 Case 3

Total recognitionTotal correctTotal recognitionTotal correct

Experiment mean 9.15 6.15 10.92 8.54

Control mean 10.53 8.27 11.27 9.27

Difference in means − 1.38 − 2.12 −.35 −.73

Common standard deviation2.31 2.37 2.56 2.78

Effect size .60 .90 .14 .26

 Case 2 Case 2 Case 3 Case 3

Total recognitionTotal correctTotal recognitionTotal correct

Power (%) 80 80 80 80

n per group 45 21 841 229

Discussion
This preliminary study compared student understanding in principles of orthognathic treatment and diagnosing and treatment planning complex orthognathic cases using the VR patient versus the conventional 2D tracing
method and evaluated feasibility of the study methods. Specifically, recruitment and retention rates, student performance on MCQ examination at pre-test, post-test and follow up time points and written responses to two
cases including scores of the responses, intra- and inter-rater reliability of the scoring, time spent on the hands-on exercise, and type of student–teacher interactions were evaluated for significance. The effect size of the
educational intervention was also investigated.

Sample description, recruitment, and retention

Third year dental students were selected as study participants because they had minimal knowledge of orthognathic treatment. They were unfamiliar with both the concepts and principles of surgical orthodontic
treatment and the conventional 2D tracing method of prediction used in treatment planning such cases. The rationale for the initial sample recruitment of 15 per group took into account "the sample size of 12 per group
rule of thumb" as described by Julious along with an estimated 20% drop-off rate of the study subjects (Julious [25]). In situations, such as this study, where there is no prior information to base the sample size and
where the result is likely a continuous outcome which takes a normal form, Julious recommended a minimum of 12 subjects be considered for pilot studies since after a sample size of 12 the gains in precision becomes
less pronounce.

In a period of 3 months, 30 students were recruited for this study. Once recruited, the retention rate of study participants was relatively high ranging from 87% in the experimental group and 100% in the control group. A
possible explanation of the differences in the retention rates may be due to different locations that the study was conducted. Experimental group had to travel 15 min by car or bus to the 3D Lab which is located in the
north part of the campus while the control group remained at the School of Dentistry. Future study should ensure that the study be conducted at the same location for both groups and the location should be convenient to
study participants.

Multiple choice question test at T1, T2, and T3 and written responses to Case 2 and 3
This study evaluated student performance at the "Knows How" (competence) level of Miller's Pyramid (Miller [31]). The participants had to demonstrate an ability to apply information learned from the lecture and hands-
on exercise to make decisions about the treatment plan for new complex orthognathic surgery cases. The MCQs were written at the "Knows How" level because students were asked to recognize the most suitable
diagnosis and treatment approach from a list of options. Additionally, the case-based MCQs assessed students at the "Apply" level of Bloom's taxonomy (Krathwohl [27]). The educational intervention which included a
lecture and hands-on exercise using the VR patient and 2D tracing methods improved student understanding of complex orthognathic surgery cases as evidence by a significant increase in the MCQ test scores
immediately following the intervention. In a previous study, higher retention in student learning using VR method was demonstrated (Krokos et al. [28]). In this study, although the VR group showed a trend in increased
retention for the follow-up test (Fig. 5), student performance at the follow-up MCQ test was not significantly different between the two groups. The difference in the study results may be due, in part, to the limited sample
size.

The novel method of scoring written responses using the Board Case Oral Examination Worksheet modified from the American Board of Orthodontics proved to be a reliable and reproducible method of quantifying
students' written responses for diagnosis, treatment objectives, and treatment plan. Future studies should utilize this method of scoring and quantifying written responses since it demonstrates excellent intra- and inter-
rater agreement.

When the percentage of mean scores of the written responses is evaluated, the students showed that they were able to recognize diagnostic criteria (experimental group: 61% vs. control group: 63%), treatment
objectives (experimental group: 49% vs. control group: 59%) and treatment plans (experimental group 71% vs. control group: 76%). The students were also able to give correct written responses to diagnosis
(experimental group: 46% vs. control group: 53%), treatment objectives (experimental group: 44% vs. control group: 54%), and treatment plans (experimental group: 36% vs. control group: 47%) for complex orthognathic
surgery cases.

Although the VR group was able to complete the worksheet more quickly than the 2D group for Case 2, there was no indication that the VR patient method was better than the 2D tracing method when evaluating student
performance. In fact, for Case 2, which was a relatively simple case showing a small lower jaw with discrepancy only in 1 plane, the students performed better using the 2D tracing method. When the students were
asked to evaluate a more complex case with discrepancy in all 3 planes of space involving both upper and lower jaws (Case 3), there was no difference between the groups in their recognition and correct responses as
well as time spent to complete the worksheet.

Effect size
Estimate of the effect size of the intervention was variable from.14 to.90 and differed greatly between Case 2 and Case 3. The effect size was large for the more simple case (Case 2); however, when the case became
more complex, as in Case 3, which included discrepancy of both jaws in all 3 planes of space, there was more variability with student responses and the effect size was small. It appears that for less complex and straight
forward cases, students are able to diagnose and treatment plan effectively using conventional 2D tracings. When the students were asked to diagnose and treatment plan a more complex case, the students were less
successful in using the 2D method. One method of improving or increasing the effect size is to decrease the variability of the students' response. A possible method of achieving this is to utilize graduate students in
orthodontics and oral surgery as the study sample rather than third year dental students. Graduate students with greater orthognathic treatment knowledge and clinical experience may show less variability and as a
result the power of the study may increase.

Additionally, one of the advantages of the VR patient is that the maxilla is presented in two pieces and the right and left sections of the maxilla can be manipulated to correct transverse discrepancies with palatal
expansion. A study cast is required for model surgery to evaluate the need for skeletal expansion for the correction of the malocclusion in the 2D tracing method. Students in both groups did not consider the transverse
dimension in their written responses. In general, third year dental students lack knowledge of complex cases involving orthognathic patients and may not have the foundational knowledge necessary to correctly diagnose
and treat such cases despite given either the 2D or the VR patient educational intervention.

Student–teacher interactions
There were significant differences in the student–teacher interactions between the two groups. The experimental group had more virtual reality (technical) interactions, such as questions related to the use of the controls
to manipulate the VR patient. This increase in student–teacher interactions can be attributed to the learning curve for the new technology rather than to problems with the diagnosis or treatment planning of the cases. It is
recommended that in future studies, instruction and training of VR hardware and software use should be conducted prior to intervention testing and data collection so that the subjects are more familiar with the virtual
reality head-mounted device and controls.

Study limitations
The ability to utilize VR allows students to visualize and interact with image data as if they were real physical objects and aid the student to understand complex spatial relationships (Mohammed et al. [33]). The VR
patient allows the user to visualize the 3D model of the patient in all planes of space to diagnose deformities and also allows the user to manipulate the model with 6 degrees of freedom. The user can perform multiple
trials to mimic several jaw surgery procedures in a virtual environment that is active and intuitive. In contrast, the 2D tracing method of jaw surgery prediction is limited to only 1 plane of space and visualization of the
entire craniofacial complex is not possible.

Although the advanced visualization and manipulation function of the VR patient software should presumably enhance the student's ability to understand concepts of jaw surgery, this study failed to demonstrate a
significant relationship between fidelity and improved outcomes in education with respect to VR. However, the results of this study should be considered cautiously given the nature of the study's small sample size and
the variability of the data. In addition to the small sample size, another limitation is related to the version of the VR patient used in this study. The experimental group was unable to simultaneously use the virtual reality
head-mounted device and provide the written responses on the worksheets. They had to continuously remove and replace the head-mounted device in order to complete the worksheet. This type of procedure clearly
disrupted the immersive experience of the VR environment. In a previous study, Gutiérrez et al. ([21]) found that students improved in fully immersed environments when compared to partially immersed environments.
Shu et al. ([45]) also detected a significant difference in pre-test and post-test measurements for earthquake preparedness and self-efficacy when using the VR head-mounted device versus desktop computer-facilitated
VR. In future studies, subjects should be prompted verbally for the response or answers should be recorded while users are in the VR scene so that the head-mounted device does not have to be removed to complete
the written responses.

This study was conducted using mouse and keyboard controls and an Oculus Rift Development Kit 2 virtual reality headset. The keyboard and mouse controls may have also limited the immersive experience of the VR
patient. At the time of the study, a more natural user interface such as hand controllers as well as the consumer version VR head-mounted device was not available. As VR hardware and software improves, the
immersive experience of the student in virtual reality should also improve and study outcomes may vary with continued advances in VR technology and further development of the VR patient software including
improvements in visualization, the ability to quantify the surgical movements within the VR scene, having the ability to record the treatment plan scenarios virtually, and using a more natural user interface.

Summary
Dental education is changing continuously with the introduction of new technological and advanced visualization innovations. This preliminary study explored the feasibility of the study methods and compared
understanding of dental students in diagnosing and treatment planning of complex orthognathic cases using the VR patient and traditional 2D tracing methods. Student understanding improved immediately following the
educational intervention; however, no significant difference was found between the two groups. For simple cases with skeletal discrepancy in one plane of space, the students appeared to perform better using the 2D
prediction tracing method. For more difficult cases involving skeletal discrepancy in both upper and lower jaws and in all 3 planes of space, there was no difference in student performance in the 2D or VR group and the
effect size of the intervention for the complex case was low.

Due to a number of limitations in this study including small sample size, disruption of the immersive experience, and limitations of the user interface, the study results could not confirm the benefit of increasing fidelity
(from 2D to VR) to help students understand principles in orthognathic patients for complex orthognathic cases involving more than one plane of discrepancy. However, based on the results of this preliminary study,
some recommendations for future full-scale tests can be made to investigate the effects of increasing fidelity (2D to 3D to VR) of the educational tools for evaluating complex orthognathic cases.

It is recommended that graduate students in orthodontics and oral surgery should be tested because they have greater knowledge of the surgical cases to answer written responses to diagnosis, treatment planning, and
surgical considerations for complex cases involving orthognathic treatment. In doing so, this may decrease the variability in the written scores and the effect size of the intervention may be increased. Separate training
sessions to familiarize the subjects with the technical aspects of VR patient and the head-mounted device and controls prior to data collection is also recommended. Verbal response to diagnosis, treatment objectives,
and treatment plan should be elicited and transcribed for the written responses so that the subjects stay engaged in the case analysis method. This study demonstrated that scoring of the written responses in diagnosis,
treatment objectives, and treatment planning is reliable, reproducible, and provides excellent rater agreement. This method of quantifying written responses should be used in future full-scale studies. Finally, VR
hardware and VR patient software improvements have continued since this study was conducted and subsequent versions are expected to be utilized for future studies.

Funding
This study was funded in part by The University of Michigan Le Gro Fund and University of Michigan Center for Research on Learning and Teaching, Faculty Development.

Acknowledgements
We thank Theodore Hall, Sean Petty, Stephanie O'Malley, Eric Maslowski, and Shawn O'Grady for developing the VR Patient module; Jason Sherbel and Justin Kammo for assisting with the research sessions; and Erin
Walker for scheduling the study participants.

Compliance with ethical standards

Conflict of interest
The authors declare that they have no competing interests.

Ethics approval and consent to participate

IRB exemption was obtained from the University of Michigan Health and Behavioral Sciences Institutional Review Board (#HUM00113004).

Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References
1 Aaker GD, Gracia L, Myung JS, Borcherding V, Banfelder JR, D'Amico DJ, Kiss S. Volumetric three-dimensional reconstruction and segmentation of spectral-domain oct. Ophthalmic Surg Lasers Imaging. 2011; 42;
Suppl: S116-S120

2 Abdelwahab MG, Cavalcanti DD, Preul MC. Role of computer technology in neurosurgery. Minerva Chir. 2010; 65; 4: 409-428

3 Abhari K, Baxter JS, Chen EC, Khan AR, Peters TM, de Ribaupierre S, Eagleson R. Training for planning tumour resection: augmented reality and human factors. IEEE Trans Biomed Eng. 2015; 62; 6: 1466-1477

4 Aebersold M, Tschannen D, Bathish M (2012) Innovative simulation strategies in education. Nurs Res Pract 2012, Article ID765212, 1–7

5 Aebersold M, Voepel-Lewis T, Cherara L, Weber M, Khouri C, Levine R, Tait AR. Interactive anatomy-augmented virtual simulation training. Clin Simul Nurs. 2018; 15: 34-41

6 Agarwal N, Schmitt PJ, Sukul V, Prestigiacomo CJ. Surgical approaches to complex vascular lesions: the use of virtual reality and stereoscopic analysis as a tool for resident and student education. BMJ Case Rep.
2012; 2012: bcr0220125859

7 Akay M, Marsh A. Information technologies in medicine. 2001: New York; Wiley

8 Arikatla VS, Tyagi M, Enquobahrie A, Nguyen T, Blakey GH, White R, Paniagua B (2018) High fidelity virtual reality orthognathic surgery simulator. Proc SPIE Int Soc Opt Eng 10576

9 Badler N, Erignac C, Liu Y. Virtual humans for validating maintenance procedures. Commun ACM. 2002; 45: 56-63

Becker O, Scolari N, Santos Melo MF, Avelar RL, Haas OLJ, Menenzes LM, Belle de Oliveira R. Three-dimensional planning in orthognathic surgery using cone-beam computed tomography and computer software.
Comput Sci Syst Biol. 2013; 6; 6: 6

Buttussi F, Chittaro L. Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Trans Vis Comput Graph. 2017; 24; 2: 1063-1076

Cao C, Cerfolio RJ. Virtual or augmented reality to enhance surgical education and surgical planning. Thorac Surg Clin. 2019; 29; 3: 329-337

Carl E, Stein AT, Levihn-Coon A, Pogue JR, Rothbaum B, Emmelkamp P, Asmundson GJG, Carlbring P, Powers MB. Virtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized
controlled trials. J Anxiety Disord. 2019; 61: 27-36

Chiarovano E, de Waele C, MacDougall HG, Rogers SJ, Burgess AM, Curthoys IS. Maintaining balance when looking at a virtual reality three-dimensional display of a field of moving dots or at a virtual reality scene.
Front Neurol. 2015; 6: 164
Dias DRC, Neto MaP, Brega JeRF, Gnecco BBG, Trevelin LC, Guimaraes MdP. Design and evaluation of an advanced virtual reality system for visualization of dentistry structures. IEEE Trans Biomed Eng. 2012.
10.1109/VSMM.2012.6365955

Dyer E, Swartzlander BJ, Gugliucci MR. Using virtual reality in medical education to teach empathy. J Med Libr Assoc. 2018; 106; 4: 498-500

Emmelkamp P, Krijin M, Hulsbosch A, de Vries S, Schuemie M, van der Mast C. Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia. Behav Res Ther. 2002; 40: 509-516

Gateno J, Xia J, Teichgraeber JF, Rosen A, Hultgren B, Vadnais T. The precision of computer-generated surgical splints. J Oral Maxillofac Surg. 2003; 61; 7: 814-817

Germans DM, Spoelder HJW, Renambot L, Bal HE, van Daatselaar S, van der Stelt P. Measuring in virtual reality: a case study in dentistry. IEEE Trans Instrum Meas. 2008; 57; 6: 1177-1184

Grabowski HA, Krempien R, Munchenberg JB, Rembold U, Worn H. Simulation of frontal orbital advancement. Virtual Reality. 1999; 4: 235-240

Gutiérrez FPJ, Vergara VM, Coulter R, Saland L, Caudell TP, Goldsmith TE, Alverson DC. The effect of degree of immersion upon learning performance in virtual reality simulations for medical education. Stud Health
Technol Inform. 2007; 125: 155-160

Izard SG, Juanes Mendez JA, Palomera PR. Virtual reality educational tool for human anatomy. J Med Syst. 2017; 41; 5: 76

Izard SG, Juanes JA, Garcia Penalvo FJ, Estella JMG, Ledesma MJS, Ruisoto P. Virtual reality as an educational and training tool for medicine. J Med Syst. 2018; 42; 3: 50

Jasinevicius TR, Landers M, Nelson S, Urbankova A. An evaluation of two dental simulation systems: virtual reality versus contemporary non-computer-assisted. J Dent Educ. 2004; 68; 11: 1151-1162

Julious SA. Sample size of 12 per group rule of thumb for a pilot study. Pharm Stat. 2005; 4; 4: 287-291

Khan R, Plahouras J, Johnston BC, Scaffidi MA, Grover SC, Walsh CM. Virtual reality simulation training in endoscopy: a cochrane review and meta-analysis. Endoscopy. 2019; 51; 7: 653-664

Krathwohl DR. A revision of Bloom's taxonomy: an overview. Theory Pract. 2002; 41; 4: 212-218

Krokos E, Plaisant C, Varshney A. Virtual memory palaces: immersion aids recall. Virtual Reality. 2019; 23; 1: 1-15

Mallari B, Spaeth EK, Goh H, Boyd BS. Virtual reality as an analgesic for acute and chronic pain in adults: a systematic review and meta-analysis. J Pain Res. 2019; 12: 2053-2085

McMillan JH, Schumacher S. Research in education: evidence-based inquiry. 20107: Boston; Pearson: 511

Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65; 9 Suppl: S63-S67

Minns S, Levihn-Coon A, Carl E, Smits JAJ, Miller W, Howard D, Papini S, Quiroz S, Lee-Furman E, Telch M. Immersive 3d exposure-based treatment for spider fear: a randomized controlled trial. J Anxiety Disord. 2018;
58: 1-7

Mohammed MAA, Khalaf MH, Kesselman A, Wang DS, Kothary N. A role for virtual reality in planning endovascular procedures. J Vasc Interv Radiol. 2018; 29; 7: 971-974

Neumann PDS, Schuiz A, Fauikner G, Krauss M, Tolxdorff T. Using virtual reality techniques in maxillofacial surgery planning. Virtual Reality. 1999; 4: 213-222

Oculus Rift CV1 Teardown (2016) https://www.ifixit.com/Teardown/Oculus+Rift+CV1+Teardown/60612. Accessed 8 Nov 2017

Oculus Rift Development Kit 2 Teardown (2014) https://www.ifixit.com/Teardown/Oculus+Rift+Development+Kit+2+Teardown/27613. Accessed 8 Nov 2017

Oculus Rift DK1 Teardown (2013) https://www.ifixit.com/Teardown/Oculus+Rift+DK1+Teardown/13682. Accessed 8 Nov 2017

Oculus Rift History—How it All Started (2015) http://riftinfo.com/oculus-rift-history-how-it-all-started. Accessed 8 Nov 2017

Orthodontics. https://www.americanboardortho.com/media/1164/bcoe-examinee-worksheet.pdf. Accessed 8 Nov 2017

Pertaub D, Slater M, Barker C. An experiment on public speaking anxiety in response to three different types of types of virtual audience. Presence Teleoper Virtual Environ. 2001; 11; 1: 68-78

Pourmand A, Davis S, Marchak A, Whiteside T, Sikka N. Virtual reality as a clinical tool for pain management. Curr Pain Headache Rep. 2018; 22; 8: 53

Proffit WR, White RP. Surgical-orthodontic treatment. 1991: St. Louis; Mosby-Year Book: 722

Robb RA. Medical imaging and virtual reality: a personal perspective. Virtual Reality. 2008; 12: 235-237

Rothbaum B, Hodges L, Smith S, Lee J, Price L. A controlled study of virtual reality exposure therapy for the fear of flying. J Consult Clin Psychol. 2000; 68: 1020-1026

Shu Y, Huang YZ, Chang SH, Chen MY. Do virtual reality head-mounted displays make a difference? A comparison of presence and self-efficacy between head-mounted displays and desktop computer-facilitated virtual
environments. Virtual Reality. 2018. 10.1007/s10055-018-0376-x

Sikka N, Shu L, Ritchie B, Amdur RL, Pourmand A. Virtual reality-assisted pain, anxiety, and anger management in the emergency department. Telemed J E Health. 2019. 10.1089/tmj.2018.0273

Smith ML, Foley MR. Transforming clinical education in obstetrics and gynecology: gone is the day of the sage on the stage. Obstet Gynecol. 2016; 127; 4: 763-767

Sweta VR, Abhinav RP, Ramesh A. Role of virtual reality in pain perception of patients following the administration of local anesthesia. Ann Maxillofac Surg. 2019; 9; 1: 110-113

Triepels CPR, Smeets CFA, Notten KJB, Kruitwagen R, Futterer JJ, Vergeldt TFM, Van Kuijk SMJ. Does three-dimensional anatomy improve student understanding?. Clin Anat. 2019. 10.1002/ca.23405

UMichDent (2019) Dentistry and VR—Jaw surgery simulation [Video File]. Retrieved from https://www.youtube.com/watch?v=YCyrqmrMwPQ. Accessed 27 Aug 2018
Uppot RN, Laguna B, McCarthy CJ, De Novi G, Phelps A, Siegel E, Courtier J. Implementing virtual and augmented reality tools for radiology education and training, communication, and clinical care. Radiology. 2019;
291; 3: 570-580

Vertemati M, Cassin S, Rizzetto F, Vanzulli A, Elli M, Sampogna G, Gallieni M. A virtual reality environment to visualize three-dimensional patient-specific models by a mobile head-mounted display. Surg Innov. 2019; 26;
3: 359-370

Walther-Larsen S, Petersen T, Friis SM, Aagaard G, Drivenes B, Opstrup P. Immersive virtual reality for pediatric procedural pain: a randomized clinical trial. Hosp Pediatr. 2019; 9; 7: 501-507

Won AS, Bailey J, Bailenson J, Tataru C, Yoon IA, Golianu B. Immersive virtual reality for pediatric pain. Children (Basel). 2017; 4; 7: 52. 10.3390/children4070052

Wong DT, Mehta A, Singh KP, Leong SM, Ooi A, Niazi A, You-Ten E, Okrainec A, Patel R, Singh M. The effect of virtual reality bronchoscopy simulator training on performance of bronchoscopic-guided intubation in
patients: a randomised controlled trial. Eur J Anaesthesiol. 2019; 36; 3: 227-233

Xia J, Samman N, Yeung R, Shen S, Wang D, Ip H, Tideman H. Three dimensional virtual reality surgical planning and simulation workbench for orthognathic surgery. Int J Adult Ortho Orthog Surg. 2000; 15; 4: 265-282

Xin B, Chen G, Wang Y, Bai G, Gao X, Chu J, Xiao J, Liu T. The efficacy of immersive virtual reality surgical simulator training for pedicle screw placement: a randomized double-blind controlled trial. World Neurosurg.
2018. 10.1016/j.wneu.2018.12.090

Zaragoza-Siqueiros J, Medellin-Castillo HI, de la Garza-Camargo H, Lim T, Ritchie JM. An integrated haptic-enabled virtual reality system for orthognathic surgery planning. Comput Methods Biomech Biomed Eng. 2019;
22; 5: 499-517

Zawy Alsofy S, Stroop R, Fusek I, Welzel Saravia H, Sakellaropoulou I, Yavuz M, Ewelt C, Nakamura M, Fortmann T. Virtual reality-based evaluation of surgical planning and outcome of monosegmental, unilateral
cervical foraminal stenosis. World Neurosurg. 2019. 10.1016/j.wneu.2019.06.057

Zheng C, Li J, Zeng G, Ye W, Sun J, Hong J, Li C. Development of a virtual reality preoperative planning system for postlateral endoscopic lumbar discectomy surgery and its clinical application. World Neurosurg. 2019;
123: e1-e8

~~~~~~~~

By Scott M. Sakowitz; Marita R. Inglehart; Vidya Ramaswamy; Sean Edwards; Brandon Shoukri; Stephen Sachs and Hera Kim-Berman

Reported by Author; Author; Author; Author; Author; Author; Author

Virtual Reality is a copyright of Springer, 2020. All Rights Reserved.