VISVESVARAYA TECHNOLOGICAL UNIVERSITY

JNANA SANGAMA, BELAGAVI-590018, KARNATAKA STATE, INDIA

A TECHNICAL SEMINAR REPORT ON

“Metaverse technology in Healthcare”

Submitted impartial fulfillment of the requirement for the award of degree of
BACHELOR OF ENGINEERING
IN
COMPUTER SCIENCE AND ENGINEERING
Submitted by

S SAINATHREDDY 1AJ20CS068

Under the guidance of

Asst Prof.G Bandhavya
Department of CSE, CITNC

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

Cambridge Institute of Technology North Campus,Bengaluru-562110
2023-2024
 CAMBRIDGE INSTITUTE OF TECHNOLOGY NORTH CAMPUS

CERTIFICATE

This is to certify that the A Technical seminar report entitled “Metaverse technology in
Healthcare” has been carried out by S SAINATH REDDY(1AJ20CS068) are bonafide

students of Cambridge Institute of Technology North Campus, in partial fulfillment of Final

year of Bachelor of Engineering in Computer Science and Engineering of the Visvesvaraya
Technological University Belagavi during the academic year 2023-2024.

Signature of Guide Signature of HOD

G Bandhavya Dr.Kavitha C
Asst.Professor, Professor and Hod,
Dept. of CSE, Dept. of CSE,
CITNC CITNC
 DECLARATION

I affirm that the technical seminar work titled “Metaverse technology in

Healthcare” being submitted in partial fulfillment for the award of Final Year Technical

seminar is the original work carried out by me. It has not formed the part of any other seminar
work submitted for award of any degree or diploma, either in this or any other University.

(Signature of the Candidate)

 ACKNOWLEDEMENT

The satisfaction that accompanies the successful completion of any task would be incomplete
without mention of the people who made it possible, whose constant guidance and
encouragement crowned our efforts with success. We take this opportunity to express my
deepest gratitude and appreciation to all those who guided us in the successful completion of this
project.

I take a great privilege to express our deep gratitude to Dr. Sendamarai P, Principal,
Cambridge Institute of Technology NC, Bengaluru for supporting us and giving us the
opportunity to carry out our project.

It is my immense pleasure to extend our thanks to our HOD, Department of Computer Science
and Engineering, Dr. Kavitha C, for providing the guidance and encouragement for the
completion of project work.

I consider it a privilege to express our thanks to our project internal guides Prof. G Bandhavya
Asst. Prof, Department of Computer Science, CIT NC, Bengaluru for her valuable suggestion
and constant encouragement, and guidance provided at every stage of our project.

Last but not the least; we would like to thank our parents, friends, teaching and non
teaching staff of CIT NC.

S SAINATHREDDY (1AJ20CS068)

i
 ABSTRACT
In recent times, the emergence of the Metaverse has garnered worldwide attention as an
innovative digital space that holds immense potential to provide a wide range of health services to medical
professionals and patients. With increasing stress on healthcare systems, it has become crucial to explore
the latest and cost-effective solutions that can provide fast and reliable medical services. The focus of this
study, therefore, is to explore applications of metaverse in various health care systems and elaborate on
how it can efficiently improve the clinical management of patients. Consequently, an in-depth assessment
of the metaverse has been carried out, while covering its core fundamentals, key technologies, and diverse
applications in healthcare and medicine, including but not limited to, emergency response learning, hands-
on experience in anatomy learning, orthopaedics, paediatrics and so on. To carry out the study, we have
used an exploratory approach to analyze qualitative data on healthcare metaverse services in our
systematic review. Relevant articles from scientific databases such as Web of Science, Springer, Scopus,
and IEEE have been identified, and the analysis has been conducted using the PRISMA reporting
guideline to ensure transparent and comprehensive reporting. The results of the study suggest that the
metaverse has the potential to transform healthcare systems by introducing novel methods for delivering
healthcare services. Metaverse's AR/VR technologies can enable remote medical consultations and
training, benefiting patients and healthcare professionals. Additionally, patients can access health-related
information and resources, empowering them to manage their health better and make more informed
decisions.

ii
 METAVERSE TECHNOLOGY OF HEALTHCARE

‘
CCC

ACKNOWLEDGEMENT i

ABSTRACT ii

LIST OF FIGURES iii

CHAPTER CHAPTER NAME PAGE

NO NO
1 Introduction 1-2
Objectives 2
Problem Statement 3

2 Related Work 4-5

3 Literature Survey 6-7

4 Methodology 8-9
Proposed System 10
Existing System 10
5 Snapshots 11

6
Requirements and 12-14
Specification
7 Implementation
15-18
Conclusion
19
References
20
 METAVERSE TECHNOLOGY OF HEALTHCARE

‘ LIST OF FIGURES
CCC

FIGURE NO TITLE PAGE NO

4 Metaverse enabling technologies 9

5 Sanpshots 10

5.1 VR accessories that provide an immersive 10

experience

5.2 Popular VR headsets A Microsoft hololens 10

2 B Oculurs Rift CVl.C HTC Vive .

6.1.1 Head-Mounted Displays(HMD) 11

6.1.2 Hand-Based Input Device 11

6.1.3 Non-Hand-Based Input Device 11

6.2.1 Scence and Object Recognition 13

6.2.2 Sound and Speech Recognition 13

6.2.3 Scene and Object Generation 13

6.2.4 Sound and Specch Synthesis 13

7.1 The Metaverse incorporates elements such 15

7.1.2 Mutimodal interaction 16

7.1.3 Multi-task interaction 16

iii
 METAVERSE TECHNOLOGY OF HEALTHCARE

‘
CCC
7.1.4 Embodied lnteraction 16

7.2 Metaverse lmplementation 17

7.2.1 Multimodal lnterface 17

7.2.2 Rl-Based Approaches 18

iv
‘
METAVERSE TECHNOLOGY OF HEALTHCARE
CCC

CHAPTER 1
INTRODUCTION

Healthcare services have stood out as a cornerstone factor in upholding the global populace's
comprehensive psychological, physical, and social well-being . The essential nature of healthcare
services is underscored by their ability to mitigate the harmful impacts of disease, injury, and
illness while simultaneously promoting wellness and longevity. However, despite the protracted
developmental journey of the healthcare industry, it has witnessed a swift and dynamic
metamorphosis, predominantly catalyzed by its integration with cutting-edge technological
advancements and the Metaverse is one such cutting-edge technological advancement that holds
the promise of better future for the healthcare industry. Presently, this sector faces many issues
and challenges, such as the widespread prevalence of chronic diseases, shortage of advanced
equipment in community hospitals, inadequate knowledge of high-end technologies, and low
patient awareness of different medical procedures, to name a few . These major impediments
have made it imperative to bring healthcare services to the doorstep of individuals . The COVID-
19 pandemic further exacerbated the situation and has been the primary impetus behind
accelerating quick change across the healthcare ecosystem. It has forced all stakeholders to
reevaluate their approach to this industry and has necessitated the unavoidable adoption of
advanced technologies to cope with the changing landscape In this scenario, the Metaverse could
prove a game changer. It represents a culmination of diverse state-ofthe-art technologies such as
Augmented Reality, Virtual Reality, Artificial Intelligence, Blockchain, Quantum Computing,
Robotics, and the Internet of Medical Devices (IoMT), among many others. By harnessing the
potential of these advanced technologies, it offers exciting opportunities for exploring innovative
solutions to deliver superior-quality healthcare treatments and services. With its potential to
provide life-like experiences to patients and healthcare providers, the Metaverse presents new
opportunities for improving healthcare outcomes. Contemporary literature suggests an increasing
interest in the metaverse and healthcare domain. However, despite numerous studies on the
implementation of metaverse technologies in healthcare, they have limitations and fail to address
certain areas. For instance, only explores potential applications of metaverse technologies in
healthcare without delving deeper into the specifics of each domain. while focuses on the
significance of Augmented Reality/Virtual Reality AR/VR technology in the healthcare sector
but does not provide enough information on how to utilize it effectively. scrutinizes the influence
of metaverse in cardiovascular medicine but lacks analysis of its potential impact on other
medical specialities. [10] advocates for the fusion of AI and blockchain in the metaverse for
2Dept .of CSE,CITNC 2023- 2024 2
‘
METAVERSE TECHNOLOGY OF HEALTHCARE
CCC
superior healthcare services, but the study lacks a practical demonstration of such a system.
identifies the areas in which metaverse technology could be used in healthcare, but the study
only reviews nine studies from South Korea and is limited in its scope.
comprehensive study is needed to examine the potential of metaverse technologies in healthcare,
identify areas that require further research, and provide actionable insights for healthcare
practitioners and companies interested in implementing such technologies. It can be concluded
that the ongoing digital revolution in the healthcare industry has gained significant momentum
due to the COVID-19 pandemic. While this shift towards telemedicine and telehealth has been
vital in providing healthcare services during the pandemic, it has also exposed limitations,
challenges, and equity concerns in the digital healthcare infrastructure. Therefore, there is an
urgent need for research on metaverse technology for healthcare systems to address these issues,
adapt to pandemic-induced changes, and meet the evolving requirements and goals of healthcare
consumers. The proposed research will systematically investigate how metaverse technology can
optimize patient satisfaction, improve medication adherence, and enhance the patient-clinician
experience by leveraging advanced digital tools and services. Furthermore, this study will
strengthen the healthcare industry's ongoing strategies to meet necessary market and technology
requirements while positioning it to respond to new business opportunity development in
metaverse services. Given the pressing need for digital healthcare infrastructure that can meet
the demands of modern healthcare, it is crucial to explore the potential of metaverse technology
in transforming and improving virtual healthcare services.

1.1 OBJECTIVES
• Use metaverse platforms to facilitate virtual doctor-patient consultations, improving
access to healthcare services.
• Develop immersive simulations for medical students and professionals to practice
procedures and learn in realistic virtual environments.
• Utilize IoT devices and virtual environments to remotely monitor patients' vital signs
and health conditions in real-time.
• Create interactive educational content and virtual support groups to empower patients
in managing their health and fostering a sense of community.

2Dept .of CSE,CITNC 2023- 2024 2

‘
METAVERSE TECHNOLOGY OF HEALTHCARE
CCC
1.2 PROBLEM STATEMENT
• Traditional healthcare systems face various challenges in delivering efficient and accessible
care to patients. These challenges include. Many individuals, especially those in remote or
underserved areas, face barriers to accessing quality healthcare due to geographical
constraints, limited resources, or a shortage of healthcare professionals. Patients often lack
sufficient education about their health conditions, leading to poor adherence to treatment
plans and suboptimal health outcomes. Additionally, there is a need for more engaging
methods to educate patients and empower them to take control of their health.
• Healthcare professionals may struggle to effectively communicate and collaborate with
colleagues, leading to delays in diagnosis, treatment, and care coordination. This problem is
particularly evident in multi-disciplinary care settings or when managing complex cases
involving multiple specialists. Medical education and training rely heavily on traditional
methods, such as textbooks, lectures, and hands-on experience. However, these approaches
may not adequately prepare students and healthcare professionals for real-world clinical
scenarios or complex procedures.
• Offering virtual doctor-patient consultations through immersive metaverse platforms,
enabling remote access to healthcare services and reducing geographical barriers. Developing
immersive educational content and virtual support groups within the metaverse to educate
and engage patients in managing their health effectively. Creating virtual spaces for
healthcare professionals to collaborate, share knowledge, and discuss patient cases in real-
time, improving communication and care coordination.
• Building interoperable metaverse platforms to aggregate and analyze healthcare data from
various sources, promoting data sharing, continuity of care, and personalized medicine. 5.
Developing realistic simulations and virtual training environments within the metaverse for
medical education, training, and procedural practice, enhancing the skills and competence of
healthcare professionals.

2Dept .of CSE,CITNC 2023- 2024 23

 CHAPTER 2
RELATED WORK
Metaverse technology and healthcare involves various initiatives and research efforts aimed at
leveraging virtual environments to improve healthcare delivery. Here are some key areas of
exploration:
• Telemedicine and Virtual Consultations: Many healthcare providers are
exploring the use of metaverse platforms to conduct virtual consultations and
telemedicine appointments. This enables patients to access healthcare services remotely,
reducing the need for in-person visits and overcoming geographical barriers. Virtual
reality (VR) and augmented reality (AR) simulations are being developed to train medical
students and professionals in realistic virtual environments. These simulations allow for
hands-on professionals in realistic virtual environments. These simulations allow for
hands-on practice of medical procedures, surgical techniques, and emergency scenarios,
enhancing training effectivenesss and safety.
• Remote Monitoring and Home Healthcare: Metaverse technology enables the
creation of immersive experiences for remote monitoring of patients' health conditions.
Through IoT devices and wearable sensors, healthcare providers can gather real-time data
and visualize it within virtual environments, allowing for timely interventions and
proactive management of chronic conditions. Virtual environments are being used to
educate patients about their health conditions, treatment options, and self-management
strategies. Interactive educational content, virtual support groups, and personalized health
portals empower patients to take an active role in their healthcare journey and connect
with others facing similar challenges.
• Data Visualization and Analysis: Metaverse platforms offer innovative ways to
visualize and analyze complex medical data, such as imaging scans, genetic sequences,
and patient records. By creating immersive data visualization tools, researchers and
clinicians can gain deeper insights into disease mechanisms, treatment outcomes, and
population health trends. Virtual meeting spaces within the metaverse facilitate
collaboration among researchers, healthcare professionals, and industry stakeholders.
Virtual conferences, workshops, and research symposiums enable the exchange of ideas,
networking, and dissemination of scientific knowledge on a global scale, while
minimizing travel costs and environmental impact. Virtual reality-based rehabilitation

CIT- NC, Dept. of CSE, 2023-2024 9

 METAVERSE TECHNOLOGY OF HEALTHCARE

programs are being developed to aid in physical and cognitive rehabilitation for patients
recovering from injuries, strokes, or surgeries. These immersive therapies engage patients
in interactive exercises and activities designed to improve mobility, coordination, and
cognitive function in a safe and controlled environment.
• Advantages:
Enhanced Accessibility: Metaverse technology enables healthcare services to reach
remote or underserved areas where traditional healthcare infrastructure may be lacking.
Virtual consultations through the metaverse offer convenience and flexibility for patients,
reducing the need for physical travel and wait times. Immersive simulations in the
metaverse allow for realistic medical training and education, offering a safe environment
for practicing complex procedures without risking patient safety. IoT devices integrated
with the metaverse enable real-time monitoring of patients' health data, allowing for early
detection of health issues and proactive intervention. Metaverse platforms facilitate
collaboration among healthcare professionals worldwide, enabling knowledge sharing,
research collaboration, and multidisciplinary consultations. Interactive virtual
environments and educational content engage patients in their healthcare journey,
promoting better understanding of their conditions and treatment options.

• Disadvantages:
Technical Challenges: Developing and maintaining metaverse healthcare solutions
requires specialized technical expertise and investment in infrastructure, which may pose
challenges for some healthcare organizations. Not all patients may have access to the
technology required to participate in metaverse healthcare services, potentially widening
existing healthcare disparities. Storing and transmitting sensitive health data in virtual
environments raises concerns about privacy and security breaches, necessitating robust
data protection measures. The use of immersive simulations and virtual environments in
healthcare raises ethical questions regarding consent, patient autonomy, and the potential
desensitization of healthcare professionals to real-life patient experiences. Integrating
metaverse technology with existing healthcare systems and workflows may be complex
and time-consuming, requiring interoperability standards and infrastructure upgrades.
While metaverse technology offers new opportunities for healthcare delivery, ensuring
the quality and accuracy of diagnosis and treatment within virtual environments remains
a challenge that requires ongoing validation and improvement.

CIT- NC, Dept. of CSE, 2023-2024 95

CHAPTER 3

LITERATURE SURVEY

• [1]R. Cheng, N. Wu, S. Chen and B. Han, "Reality Check of Metaverse: A First Look at
Commercial Social Virtual Reality Platforms," 2022 IEEE Conference on Virtual Reality
and 3D User Interfaces Abstracts and Workshops (VRW), 2022, pp. 141-148, doi:
10.1109/VRW55335.2022.00040. Metaverse has grasped the news headlines recently.
While being heavily advocated by the industry, there are great interests from academia as
it demands various technological support from both hardware and software research.
There has been an extensive visionary discussion of metaverse lately, but there are few
studies on its technical challenges and user experience in practice. To fill this critical gap,
in this paper, we take a first look at Workrooms, arguably a premature metaverse product
released by Meta (a.k.a. Face-book). [2] S. -M. Park and Y. -G. Kim, "A Metaverse:
Taxonomy, Components, Applications, and Open Challenges," in IEEE Access, vol. 10,
pp. 4209-4251, 2022, doi: 10.1109/ACCESS.2021.3140175. With the technological
development of deep learning-based high-precision recognition models and natural
generation models, Metaverse is being strengthened with various factors, from mobile-
based always-on access to connectivity with reality using virtual currency. The
integration of enhanced social activities and neural-net methods requires a new definition
of Metaverse suitable for the present, different from the previous Metaverse. This paper
divides the concepts and essential techniques necessary for realizing the Metaverse into
three components (i.e., hardware, software, and contents) and three approaches (i.e., user
interaction, implementation, and application) rather than marketing or hardware approach
to conduct a comprehensive analysis. [3] S. Tayal, K. Rajagopal and V. Mahajan, "Virtual
Reality based Metaverse of Gamification," 2022 6th International Conference on
Computing Methodologies and Communication (ICCMC), 2022, pp. 1597-1604, doi:
10.1109/ICCMC53470.2022.9753727. The Metaverse is a podium of user mesmerizing
interactions in the virtual world with the support of the Internet, software, and hardware
required to create the exceptional immersive user experience. Gamification is becoming
part of our life, and with the growing technology and digitalization, it has become a
prevalent part of different user activities. The increasing reputation of Metaverse is
opening the more user immersive online interaction in the virtual internet world where
users can meet, participate, and collaborate for a specific purpose such as social
interaction. [4] K. Getchell, I. Oliver, A. Miller and C. Allison, "Metaverses as a Platform

CIT- NC, Dept. of CSE, 2023-2024 9

METAVERSE TECHNOLOGY OF HEALTHCARE

for Game Based Learning," 2010 24th IEEE International Conference on Advanced
Information Networking and Applications, 2010, pp. 1195-1202, doi:
10.1109/AINA.2010.125. This paper evaluates metaverses as a platform for game based
learning. Metaverses such as Second Life are a relatively new type of Internet application.
Their functionality is similar to that offered by 3D multi-player online games, but differs
in that users are able to construct the environment that avatars inhabit and are not
constrained by predefined goals of the type found within a game environment. From a
quality of service (QoS) perspective metaverses are similar to games in that the timeliness
network of communication is important, but differ in that their demands upon host server
systems and network traffic are more bandwidth intensive. This paper contributes to our
understanding of metaverses by presenting a case study of the application of Game Based
Learning (GBL) within a metaverse environment, by situating the case study within a
survey of the state of the art in GBL in metaverses and by analysing the QoS delivered
by the widely used Second Life metaverse under a range of evaluator-induced network
conditions.

CIT- NC, Dept. of CSE, 2023-2024 97

 CHAPTHER 4
METHODOLOGY
• Implementing metaverse technology in healthcare involves a systematic methodology to
ensure successful integration and utilization. Here's a basic methodology:
• Needs Assessment: Identify healthcare challenges and areas where metaverse technology
can provide solutions. Determine the specific needs of patients, healthcare providers, and
organizations.
• Research and Development: Invest in research and development to explore the feasibility
and potential applications of metaverse technology in healthcare. Collaborate with experts
in both fields to develop innovative solutions.
• Platform Selection: Choose appropriate metaverse platforms or develop custom solutions
tailored to healthcare needs. Consider factors such as scalability, interoperability,
security, and user experience.
• Pilot Testing: Conduct pilot tests or trials to evaluate the effectiveness and usability of
metaverse healthcare solutions. Gather feedback from users and stakeholders to iterate
and improve the technology.
• Integration with Existing Systems: Integrate metaverse technology with existing
healthcare systems, such as electronic health records (EHRs) and telemedicine platforms,
to ensure seamless interoperability and data exchange.
• Training and Education: Provide comprehensive training and education programs for
healthcare professionals, patients, and other stakeholders on how to use metaverse
technology effectively and safely.
• Regulatory Compliance: Ensure compliance with relevant healthcare regulations and
standards, such as HIPAA (Health Insurance Portability and Accountability Act) in the
United States, to protect patient privacy and security.
• Scaling Up: Gradually scale up the implementation of metaverse technology across
different healthcare settings, departments, and regions. Monitor performance metrics and
adjust strategies as needed.
• Continuous Improvement: Continuously monitor and evaluate the impact of metaverse
technology on healthcare outcomes, patient satisfaction, and operational efficiency.
Iterate and improve the technology based on feedback and data analysis.
• Collaboration and Partnerships: Foster collaboration with technology companies,
research institutions, healthcare organizations, and government agencies to drive
innovation and address complex healthcare challenges using metaverse technology.

CIT- NC, Dept. of CSE, 2023-2024

 METAVERSE TECHNOLOGY OF HEALTHCARE

Fig4. METAVERSE ENABLING TECHNOLOGIES

CIT- NC, Dept. of CSE, 2023-2024 9

 METAVERSE TECHNOLOGY OF
HEALTHCARE

4.1 EXISTING SYSTEM

Patients can virtually meet healthcare providers in immersive environments, enabling remote
consultations akin to in-person visits. This application utilizes VR or AR headsets, allowing physicians
to diagnose, prescribe treatments, and monitor patients from a distance. Medical professionals can
engage in lifelike simulations of surgeries, medical procedures, and emergency scenarios within the
metaverse. This offers a safe environment for practice and skill development, potentially reducing
medical errors and improving patient outcomes. Complex medical data, such as imaging scans or
genomic sequences, can be visualized and analyzed in three dimensions within the metaverse. This
facilitates deeper insights into diseases, treatment responses, and personalized medicine
approaches.IoT devices and sensors can transmit real-time health data to virtual environments, where
healthcare providers can monitor patients remotely. Additionally, caregivers or family members can
virtually "be present" with patients even if physically distant, enhancing social support and reducing
isolation. Patients can access immersive educational content to better understand their health
conditions, treatment options, and self-care strategies. Virtual support groups and communities provide
opportunities for peer support and shared experiences, fostering empowerment and resilience. Virtual
environments offer customizable rehabilitation programs for physical and mental health conditions.
Patients undergoing rehabilitation can engage in interactive exercises, therapies, and cognitive training
activities within immersive simulations tailored to their needs.Researchers and healthcare professionals
can collaborate globally within virtual environments, sharing data, conducting experiments, and co-
designing innovative solutions. This fosters interdisciplinary collaboration, accelerates scientific
discovery, and enhances the translation of research findings into clinical practice. Metaverse healthcare
solutions should prioritize accessibility for individuals with disabilities, ensuring that virtual
environments are navigable and usable for all users. This involves considering factors such as visual
impairment, motor limitations, and cognitive disabilities in the design and implementation of
immersive technologies.

4.2 PROPOSED SYSTEM

Features include real-time video conferencing, medical data visualization, and interactive tools for
diagnosis and treatment planning. Medical students and professionals engage in immersive
simulations to practice procedures and enhance clinical skills. Virtual environments replicate
hospital settings, enabling learners to experience realistic scenarios and emergencies. Inpromoting
engagement and adherence to treatment plans.
CIT- NC, Dept. of CSE, 2023-2024 10
 CHAPTER 5
SNAPSHOTS

Fig 5.1 VR accessories that provide an immersive experience

5.2 Popular VR Headsets. A Microsoft Hololens 2 B.Oculurs Rift CVI C.HTC

Vive D. Oculurs Quest 2

CIT- NC, Dept. of CSE, 2023-2024

 CHAPTER 6

REQUIREMENTS AND SPECIFICATIONS

Hardware Requirements:
• Hardware in Metaverse not only plays an important role in the immersive experience but also is a
technically limiting barrier. In the Metaverse, hardware is quickly enhanced by the effects of
technological advancement, but it still needs improvement compared to the experience of the real
world. The essential hardware of Metaverse is an HMD that blocks the view to enable immersive
participation.
6.1.1 Head-Mounted Displays (HMD)
• The HMD shows an image through the display and plays the role of playing the sound through the
speaker. a basic input tool of Metaverse and is divided into Non see-through HMD, Optical-see-
through HMD, and video see-though HMD. In the case of a method that covers the screen, it
provides a sense of immersion in a completely virtual world. Optical-see-through (mainly used in
AR) is a method of overlaying the virtual world, and high hardware specifications are required in
the process of overlaying. To complement this method, video-see-though HMD is used. These
HMD issues are the bulky, expensive, and short battery life of the headset. HMD tracks position
and orientation according to the movement of the head and delivers the same change of view as in
the virtual world by moving the screen. It is more inaccurate than the method of estimating motion
by external measurement due to problems with accuracy and delay time, but it is widely used
because it can save space and cost.
6.1.2 Hand-Based Input Device
• Diverse circular coordination and input area are proposed for hand-based input devices. Detailed
user data modelling (e.g., mobile phone grip prediction) is required to provide feeling the material
with tactile. Haptic has a passive haptic that gives the texture of real objects and an active haptic
that creates virtual pressure. Passive haptic is used to help understand the situation while giving
presence, and active haptic is used for more effective interaction by adjusting and delivering
according to user feedback. Using real props (e.g., physical degree and operational degree) in a
virtual environment helps the user experience, while using a robotized interface allows for more
diverse interactions. Depending on the device’s installation, it is divided into the case of being
attached to the hand and the case of being attached to the outside. Beyond making the material
feel, it is used in various forms (e.g., inducing muscle tension).
6.1.3 Non-Hand-Based Input Device
• As auxiliary input means, there are eye-tracking, head tracking, voice input device, and so on.
Eyetracking is a method of changing the viewpoint by predicting eye movement when the user
moves their eyes without turning their heads. It is a technology that allows the user to see what
kind of object the user is paying attention to. It has the advantage of reducing the load on image
processing by generating high-resolution images in the section where the user is focused on a
phobia method. The method of overlaying the display on the arm is more stable than the method
in the air by repeatedly providing the display at a location predictable by the user. Voice input has
an advantage in processing long texts and conversations in a virtual keyboard and an environment
where input is limited.

CIT- NC, Dept. of CSE, 2023-2024

 METAVERSE TECHNOLOGY OF HEALTHCARE

Software Requirements:

A cognitive illusion plays an essential role in immersion in the objective reality of the physical
space and the subjective reality that users feel. There are two types of cognition: static cognition
and dynamic cognition. Static cognition is the proprioceptive senses (e.g., sight, hearing, and
touch), while dynamic cognition is sensory balance and body movement. In dynamic cognition,
adaptation, attention, and behaviour are important features. According to the object of cognition,
it can be divided into the cognition of environment and cognition of an object. In particular, in
Metaverse, it is important to reduce the distortion of detection and recognition. Methods for
mitigating distortion include changing the shape of the kernel, changing the expression, and
increasing the input. Objects of object recognition include faces, poses, gestures, and gazes related
to the body. Such object recognition goes through the process of sensing, recording, recognizing,
and tracking. There are two types of stimulation: remote and proximity stimulation. There are
bottom-up and top-down approaches to perceiving stimuli. A concept of perception that is distinct
from this intuitive sense is also needed. The unconscious approach and the conscious approach are
classified according to the presence or absence of a difference in movement according to repetitive
recognition. There are instinctive, behavioural, contemplative, and emotional processing methods.
The avatar is an important entity in the Metaverse, and the avatar is created, and the action is
imitated using animation. Vision-based models estimate human poses, recognize hand gestures
and predict gaze. To predict the gaze, iris, facial contour, and 3D gaze prediction are used.

Adoption of testing frameworks (e.g., pytest) for automated unit testing, integration testing, and
regression testing.
Integration with continuous integration (CI) tools (e.g., Jenkins, Travis CI) for automating testing,
code quality checks, and deployment processes.

6.2.1 Scene and Object Recognition

Object recognition is the process of recognizing the size, shape, position, brightness, and colours
of objects according to distance. For scene recognition and object recognition, novel methods
(e.g., modal alignment, cross-modal attention, point cloud, and scene graph) are used. Scene
recognition is a good recognition of what state the current scene is and what components and
configurations it has. In sub-graph-based scene graph generation, a method of clustering object
pairs into graphs by clustering and sharing representations is used. Scene graphs are a good
approach to complement the explainable properties that have emerged as limitations of neural
network models. Some studies use generative methods and scene graphs to classify bodies in
overlapping situations and predict human postures behind walls. Object recognition is also
important along with scene recognition, and we have to pay attention to human-centred scene
analysis and non-contact interaction (e.g., gaze, gesture, pose). When many objects are
recognized using individual object detection, the number of computations increases in proportion
to the number of objects, so an attempt is made to reduce the computational burden by using an
abstraction concept. In particular, some studies (e.g., world models and MONET) abstract
multiple objects into representations for fast object recognition and efficient training.

CIT- NC, Dept. of CSE, 2023-2024 13

METAVERSE TECHNOLOGY OF HEALTHCARE

6.2.2 Sound and Speech Recognition

• Recognizing sounds and processing speech help understand surroundings and
communicate with other avatars. The conversation is a direct method of communication
with other avatars and giving instructions to NPCs in Metaverse. As the Metaverse
connection is made in various environments, it is necessary to have a technology that
separates the surrounding noise and one’s own voice without noise. In addition, the
loudness of the sound according to the distance is a variable. For a realistic environment
in the Metaverse, voice recognition technology is needed that considers the surrounding
environment while adjusting the volume according to the distance.
6.2.3 Scene and Object Generation
• The method of generating the environment and objects in Metaverse is divided into the
method of depicting by reflecting the real world and the method of creating a new
imaginary environment. A realistic way to reflect the real world environment is to
reproduce famous places (e.g., museums, Eiffel Tower) and places familiar to individuals
(e.g., home, school) in the real world. Alternatively, it creates a hard-to-reach
environment (e.g., underwater, Mars) to provide a surreal experience. People and things
are the main objects of object generation. Object generation modules create an avatar and
NPC of any desired human shape (e.g., a celebrity, a family member) as an object of
conversation. It focuses on facial expressions and natural movements of joints for fluent
multimodal conversation. On the other hand, it generates realistic objects that express in
detail enough to feel the texture of objects that exist in reality. On the other hand, another
type of object is imaginary animals (e.g., unicorns, dragons) and anthropomorphic objects
(e.g., talking chairs) that do not exist.
6.2.4 Sound and Speech Synthesis
• Sound synthesis is a field that gives the user a sense of immersion, but research is
insufficient compared to vision. It creates a sound in the space to give a feeling of
presence in the field and to increase the sense of immersion. In particular, a voice suitable
for each character is an important means of expressing the character’s persona. Taco Tron,
a speech synthesis, focuses on that users can use prosody to emphasize words or express
uncertainty. Prosody is the variation of the speech signal that remains after taking the
variation into account (e.g., phonetics and channel effects), which captures meaningful
utterances and transfers them by subtractive methods.
6.2.5 Motion Rendering
• CNNs and global context encoding are used to capture asymmetric dependencies and
context patterns between objects in real-time multi-party 3D motion capture and pose
estimation. The graph reflects the structural characteristics of the body to interpret the
action meaning more accurately when the human body is superimposed. Although it is
possible to capture the real-time 3D motion of difficult scenes with a single-color camera
and isolate human body structures (e.g., shaking hands), it is still limited in capturing
close interactions (e.g., hugs)
CIT- NC, Dept. of CSE, 2023-2024 14
 CHAPTER 7

IMPLEMENTATION

7.1 USER INTERACTION

• Natural interaction is an essential condition for increasing immersion in the Metaverse. It
can reproduce the faces of friends and celebrities to enable realistic interactions and to
instil the illusion of users with familiar and famous places. Temporary dissociation,
concentration, and heightened enjoyment are important factors in the interaction, and
emotions of control, curiosity, and intrinsic motivation are used. The target of interaction
is mainly human, and hands are an important feature. Input devices are broadly divided
into handheld devices and non-hand input devices. Fidelity, proprioception, and
egocentric view are important for interactions on physical devices. Since a 360-degree
field of view is used as the receptive field for spatial recognition, a lot of images and
distortion corrections are required for video processing efficiency. In order to reduce
motion sickness and fatigue, visual and bodily sensory collisions and an alternative
sensory method are needed. It also requires multimodal sensory perception that handles
speech, gestures, and dialog flows.

Fig 7.1 The Metaverse incorporates elements such as Privacy security interaction and interface.

CIT- NC, Dept. of CSE, 2023-2024

17
16
 METAVERSE TECHNOLOGY OF HEALTHCARE

7.1.2 Multimodal Interaction

• Humans facilitate efficient adaptation and reason more abstractly by transferring
knowledge across tasks. People communicate not only dialogue but also based on
multimodal information (e.g., facial expressions, gestures, and tone of voice). The method
of handling each modal is difficult to handle multiple complex emotions, so multimodal
interaction is required. In general, multimodal has more information than unimodal and
is advantageous for understanding the situation. Text and images in social media posts do
not have the same meaning but instead have more complex meanings that intersect
semantically. In particular, multimodal learning is most effective when the meanings of
images and text are different. After the advent of Transformers, studies have been
conducted to learn vision and language together and reduce learning from scratch using a
pre-trained model.

7.1.3 Multi-Task Interaction

• Since the Metaverse handles many things in the cyber world, a model that handles
multiple tasks simultaneously is useful in the aspect of complexity. For such a model,
knowledge distillation is used to make a small model that performs many functions and
handles other modal types (e.g., Visual QA). It is relatively easy to use for similar tasks
but easily overfits when target domain data is scarce and has a different distribution. E2E
methods are also used to perform various tasks effectively. Translatotron translated from
voice input to voice output through a sequential process. Compared to the cascaded
model, the E2E model has the advantage that most of the inputs can be utilized without
data loss in the process. Translatotron interprets a foreign language, including its unique
pronunciation and emotional meaning. Also, it has the advantage of responding in a voice
form that reflects the prosody of the actual speaker. method SLU for a cloud-based
modular dialog system (SDS), showing that it is effective in situations with low ASR
accuracy

7.1.4 Embodied Interaction

• The difference between the Metaverse and other general interactions is that the proportion
of embodied interactions (e.g., embedded QA and visual language navigation) is
relatively high. While the required skills are similar to EQA and VLN, there is a
difference in whether the subject is active or passive. While the purpose of VQA is to
answer text questions about a given image, EQA (embodied question and answer)
performs the task of analysing sensor information obtained by an agent materialized
through active exploration. For example, to answer a question about the colour of a car at
a distance, the agent actively moves, recognizes, and responds based on prior knowledge
of the car’s location and path. These EQA tasks have recently been extended in the form
of conversations, where agents compensate by querying oracles for insufficient
CIT- NC, Dept. of CSE, 2023-2024 16
 METAVERSE TECHNOLOGY OF HEALTHCARE

information to perform the task. The factor that differentiates embedded interaction from
2D-based methods is Exophora resolution. People communicate information in a non-
verbal form by pointing to an object instead of language. When a user points to a specific
location through a finger, it becomes an intended instruction. In the case of exophora
resolution, specific instructions are performed in terms of multimodal interaction,
including motion and speech, whereas anaphora simply links meaning between texts.

7.2 METAVERSE IMPLEMENTATION

• The process of Metaverse implementation is divided into a design phase, a model-training
phase, an operation phase, and an evaluation phase. The design phase considers goals and
concept design, development time and cost, risk estimates, constraints, user scenarios,
scope and requirements, and feasibility of implementation and evaluation. In the model
training phase, data analysis, user modelling, scientific methodology, iterative learning,
and parameter tuning are performed. The operation phase considers system
considerations, simulations, job scheduling, network environments, and prototype
demonstrations. The evaluation phase deals with content fidelity, the authenticity of
interactions, implementation feasibility, and failover. This survey covers three types of
multimodal inference, RL-based approaches, and lifelong learning for Metaverse training
models. In addition, it is necessary to consider multiagent optimization, integration
optimization, and operational considerations from the perspective of Metaverse service
operation.

7.2.1 Multimodal Inference

• Humans do not only interpret the meaning of utterances when communicating with others.
When information is given from the cognitive model, it interprets its meaning, combines
it with its knowledge, and inferences its intentions. Verbal ambiguity is compensated to
determine the speaker’s underlying intentions based on direct or indirect representations
of the surrounding environment. For example, emotion recognition, the initiator of
emotional interaction, uses multimodal fusion to compensate for the lack of context in
textual information. Multi-modal models do not always outperform single-mode models,
so they should be utilized according to the situation. Recently, Dialog GPT and Vlbert
are proposed to implement dialog and visual-language tasks more conveniently.
Largescale pre-trained language models (PLMs) (e.g., Bidirectional Encoder

CIT- NC, Dept. of CSE, 2023-2024

17
 METAVERSE TECHNOLOGY OF HEALTHCARE

Representations from Transformers (BERT), GPT-3) are used for downstream tasks by

applying finetuning and few-shot learning.

7.2.2 Rl-Based Approaches

• Multi-agent RL, Imagination-augmented RL, and Language grounded RL are utilized in
Metaverse because RL is suitable for action in a situation without prior learning. Multi-
agent RL provides realistic NPCs by causing collaboration and disputes among various
agents. Imagination-augmented RL has the feature of rapidly stabilizing without
enormous training data, and language-based RL is used for conversation. Technically, RL
is a method to achieve an objective goal by determining the behaviour that will receive
the maximum reward based on the state received from the environment. It is divided into
model-based RL and model-free RL according to the existence of a model for a task. It is
also divided into a value-based method and a policy-based method according to the
training method. The on-policy method trains an algorithm using the deterministic output
of the target policy, whereas the off-policy method indirectly creates and trains a stored
distribution. Compensation methods (e.g., episodic memory, world model, and language-
based RL) have been proposed to solve the problem of inefficiency and sparse rewards of
RL sampling. Furthermore, more efficient approaches (e.g., offline RL and control RL)
are emerging to solve fundamental problems (e.g., sample inefficiency, unstable training).
Unlike traditional off-policy RL and model-based RL, offline RL uses only precollected
training data, not online results. Offline RL shows reliable learning with batch training
and good performance in a closed-loop environment. RL methods are steadily growing
through knowledge sharing, memory, abstraction, and language bases. The Diversity all
you need (DIAYN) model learns useful skills without a reward function, just as humans
navigate the environment without supervision. DIAYN acquires skills by maximizing
information-theoretic goals using a maximum entropy policy.

7.2.3 Life-Long Learning

• Life-long learning is meaningful because it builds experience points over a long period in
a sustainable Metaverse. For such life-long learning, a method that effectively memory
existing data and use it a an appropriate time is required. Most solutions and services have
a constant cycle. In order to apply lifelong learning to Metaverse, it is necessary to
consider how to maintain long-term service. On the other hand, the key to life-long
learning is how to handle the catastrophic forgetting that most neural net models have.

CIT- NC, Dept. of CSE, 2023-2024

18
 CONCLUSION
The healthcare industry faces challenges due to limited resources, funding constraints, and
geographic barriers that hinder access to care. Leveraging the capabilities of the metaverse in
healthcare can lead to cost reductions, redirecting resources to areas like advanced screening
technologies. The global healthcare market in the metaverse is expected to reach 71.97 billion
dollars by 2030 [2], reflecting significant growth potential. This presents an opportunity to
transform the industry, improving access, cost-effectiveness, and patient care. To this end, the
undertaken study provides a comprehensive evaluation of academic discussions and research
related to the Metaverse, the healthcare industry, and emerging technologies. Its objective is to
explore how the Metaverse can revolutionize healthcare in the future. To establish a strong
foundation, the study delves into the fundamentals of the Metaverse and its potential applications
in various medical domains. These applications range from enhancing medical education and
training to enabling immersive clinical care, performing surgical procedures, supporting
obstetrics and so on. The shifting mindset, characterized by the increased adoption of telehealth
and the influence of gaming on younger generations, reflects a growing acceptance of technology
in healthcare, as has been revealed by the study. As we strive for proactive healthcare,
technologies like virtual reality (VR), blockchain, and digital twins play a significant role in
transforming the healthcare landscape. However, while recognizing the potential of the
Metaverse in improving healthcare, it is vital to approach its implementation with a focus on
preserving the quality of care and ensuring patient safety. Moreover, it is crucial to acknowledge
and address the impact on the human dimension of healthcare when integrating technology. The
ethical and practical considerations surrounding data security and privacy, interoperability
challenges, and legal complexities associated with data ownership and distribution must be
carefully navigated. By proactively addressing these challenges, we can ensure that the
implementation of Metaverse systems aligns with the principles of quality care and patient safety.
By emphasizing a patient-centered approach and maintaining a comprehensive understanding of
the potential risks and benefits, we can effectively harness the power of the Metaverse in
healthcare. This requires a thoughtful and conscientious approach, where technology acts as an
enabler rather than a disruptor. Through this approach, we can maximize the benefits of the
Metaverse in improving healthcare outcomes while safeguarding the well-being and interests of
patients.

CIT- NC, Dept. of CSE, 2023-2024

 27

CIT- NC, Dept. of CSE, 2023-2024

 REFERENCES

G. Salloum and J. Tekli, "Automated and personalized nutrition health assessment, recommendation, and
progress evaluation using fuzzy reasoning," International Journal of Human-Computer Studies, vol. 151, p.
102610, 2021.

1. R. Chengoden et al., "Metaverse for Healthcare: A Survey on Potential Applications, Challenges and Future
Directions," IEEE Access, 2023.
2. Y. Lu, D. Yang, Y. Yang, and C. Bai, "MIoT integrates health, MM benefits humans: Funding conference
for International Association and Alliance of Metaverse in Medicine successfully held," Clinical eHealth,
vol. 5, pp. 17-18, 2022.
3. M. Alshamrani, "IoT and artificial intelligence implementations for remote healthcare monitoring systems:
A survey," Journal of King Saud University-Computer and Information Sciences, vol. 34, no. 8, pp. 4687-
4701, 2022.
4. Kapoor, S. Guha, M. K. Das, K. C. Goswami, and R. Yadav, "Digital healthcare: The only solution for
better healthcare during COVID-19 pandemic?," vol. 72, ed: Elsevier, 2020, pp. 61-64.
5. T. Shakeel et al., "A survey on COVID-19 impact in the healthcare domain: worldwide market
implementation, applications, security and privacy issues, challenges and future prospects," Complex &
intelligent systems, pp. 1-32, 2022.

CIT- NC, Dept. of CSE, 2023-2024