2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)
Design and Implementation of Virtual-real Interactive System for Mixed Reality
Xinxin Shi, Xin Xiang, Long 濬濸澳
Key Laboratory of Media Audio & Video (Communication University of China)澿 Ministry of
Education
sxx@cuc.edu.cn, xiangxin@cuc.edu.cn, yelong@cuc.edu.cn
Abstract
Mixed reality (MR) combines real and virtual worlds
to create an interactive bridge between the real world, the
virtual world and users. In order to make the interaction
between the user and the virtual world intelligent, a
mixed reality two-way virtual-real interaction system
scheme is proposed. This paper mainly discusses the
technical solutions for implementing the system and
demonstrates the interaction between the user and virtual
characters in the interactive system. The virtual
characters of the interactive system are stimulated by real
and virtual scenes, and the behavior of real people
changes [1]. The modules used in this study include: 1)
Unreal Engine 4 builds virtual scenes and characters, 2)
Motion capture system captures real-world information,
aligns real and virtual worlds. Alignment refers to the
same position information and orientation information of
the real world and the virtual world. 3) Graphics
workstation synthesizes scenes captured by optical
cameras and virtual scenes.
Keywords: Mixed reality, Unreal Engine 4, Chroma key,
Optical tracking
1. Introduction
MR is a further extension of virtual reality, combining
real and virtual worlds. Mixed reality combines virtual
scenes with the physical world to form a coherent reality
[2]. In the new visual environment, the real and virtual
worlds coexist, and the user can manipulate the virtual
objects by hand to interact [3]. The key points of MR are
real-time acquisition and interaction of virtual world
information in the real world. Interaction means that
communication and communication between real people
and virtual reality systems can be achieved.
In this paper, the concept of real and virtual objects
interaction is proposed [4]. In the MR space, the position
information of the objects can be acquired in real time,
and the collision of the virtual world object can be
monitored in real time, and different trigger scenes can be
set for different collisions.
978-1-7281-1198-8/19/$31.00 ©2019 IEEE
DOI 10.1109/MIPR.2019.00096
The purpose of the mixed reality two-way virtual
reality interaction system is to achieve interactions
between physical and virtual objects (human) within the
mixed space, including spatial alignment and interaction
coherence. The system consists of multiple modules,
including capturing real scenes in defined spaces, as well
as virtual environment rendering components and image
composition components. We first introduce the entire
system and then introduce more details [2]. The
interactive system framework is shown in Figure 1.
Figure 1. System framework
2. Module Composition of System
Virtual reality (VR) means a completely immersive
experience that shuts down the physical world.
Augmented Reality (AR) typically adds digital elements to
a live view by using a camera on a smartphone. MR
combines the elements of AR and VR, combining real
world and digital objects. The lightweight 3D engine -
Unreal Engine 4 (UE4) can quickly create virtual scenes
for MR. UE4 has a large number of plug-ins in the
direction of setting up a virtual scene, which can quickly
set up a scene [5].
UE4 is responsible for creating virtual scenes,
including virtual objects (people) and collision monitoring
and interaction design. Motion Capture tracks the real
objects (people) in the mixed space, analyzes human
motion [6], and is responsible for the combination of real
and virtual worlds. The graphics workstation is
responsible for merging the unreal scenes with the real
scenes captured by the optical tracking camera.
2.1. Unreal 濙ngine 4 & 濣ptical tracking
475
 In this MR interactive system, real-time interactive
display is an important part of it. UE4 is the framework of
the entire system for scheduling a series of subsystems
such as real-time rendering, sound effects, animation, and
physical collision detection [7]. Because UE4 is capable
of high-efficiency, high-quality real-time rendering, and
has the ability to open hardware interfaces, it has a high
degree of fit with the needs of MR. In the real-time
rehearsal for interaction, UE4 can meet the requirements
of high-definition, real-time data acquisition and motion
capture module integration in real-time interactive
rehearsal, which makes it have high platform value in MR
production.
MR has the following special requirements for the
rendering platform: 1) High-quality 3D scene real-time
rendering capabilities. 2) Open hardware and software
interface, easy to integrate third-party data, remote control.
3) Abundant editable capabilities, including scene editing, Figure 3. Hand tracking mark
camera editing, lighting editing, etc. 4) Virtual platform
conversion, import and export between multiple platforms. 2.2. Chroma key
UE4 builds a virtual scene and needs to align it with
the real scene. This research uses the action capture MR is a combination of real world and digital objects.
system made by Optitrack (https://www.optitrack.com/) to It is necessary to fuse real and illusory scenes. Digital
align the real world with the virtual world and fix their image synthesis is divided into two methods: masking
relative coordinates. The system consists of 6 tracking image synthesis and Chroma key image synthesis.
cameras, 2 × 2m capture area. The structure of Optitrack Image synthesis can increase the picture capacity and
motion capture system is shown in Figure 2. enhance the picture expression. Different materials are
placed on different layers and combined into a new
picture [10].
The chroma key is a color selection method for image
switching, which is mainly used for late image processing
of a lens shot on a blue screen or a green screen. The
person being photographed is usually arranged in front of
the green background, and the late green is completely
excluded from the subject. Where the screen displays
green, it is replaced with a background image resource.
The chroma key is convenient and fast, so this experiment
uses the chroma key. The chroma key image synthesis
structure is shown in Figure 4.

Figure 2. Optitrack motion capture system Figure 4. Chroma key structure

Optitrack can capture the movement of objects, and
reflective markers are attached to the moving objects [8].
The system draws the environment coordinates, traces the
marked objects and calculates the relative position [9].
The hand tracking markers for this study are shown in
Figure 3.
In this experiment, the real person is photographed in
front of the green screen, and the foreground image is
transmitted to the graphic workstation, and the image is
separated from the green screen and superimposed with
the background image to finally obtain the superimposed
image. Figure 5 is a screen shot by the camera. Figure 6 is
a picture of camera combined with the virtual scene.

476

Figure 5. Picture taken by the camera
Figure 6. Synthesized picture
4. Two-way virtual-real interaction system
construction
The graphics workstation can combine the video
acquired by the optical camera with the virtual scene.
Next, UE4 is used to design the interaction process.
This virtual-real interaction system consists of two-way
interactions, consisting of two steps: the real person
passes the object to the virtual character, and the virtual
character passes the object to the real person. With the
477
Optitrack plug-in, the virtual object is bound to the hand
markers in UE4, so that the two can share the position
information and synchronize the displacement. Figure 7 is
a picture of a virtual object bound to a hand.
Figure 7. Virtual object binding to the hand
1ȻReal character passes objects to virtual character
The UE4 shares the position information with the real
object, and the virtual object moves by the movement of
the hand. In the UE4, the collision between the virtual
object and the character can be monitored. When the two
collide, different collision effects can be preset according
to different collision types [11]. In this experiment, a bowl
is delivered. The virtual character can reach out to grab
the bowl according to the position of the bowl. If the
virtual character gets the bowl, he can make a drinking
action. Figure 8 is collision bodies on a virtual character.
Figure 9 is an action after the avatar listens to the collision.
Figure 10 is a picture of the virtual character gets a virtual
object. Figure 11 presents the drinking action made by the
virtual character after he gets the bowl.
Figure 8. Collision bodies on a virtual character
 Figure 9. An action after the avatar listens to the
collision
Figure 10. Virtual character gets virtual object
Figure 11. Virtual character action
2ȻThe virtual character passes the object to the real
character
The real character's hand moves continuously, and the
UE4 requests the Optitrack for the location information of
the hand markers. Then the virtual character passes the
object to the position of the markers. When the object is
transferred to that location, the real person grabs the
object. The delivery process is shown in Figure 12.
478
Figure 12. Delivery process
5. Conclusion and future work
MR combines real and virtual scenes extends the real
world and allows users to interact with digital information
or objects in the real world. This paper has proposed a
mixed reality two-way virtual-real interactive system and
implements the system using related technologies. The
system uses UE4 as a 3D engine, which makes the
background scene construction more flexible and
extensible. In addition, it uses Optitrack as motion capture
system, and UE4 provides plug-in support, which matches
well and simplifies the introduction of technology. The
feasibility of this technology has been proven
experimentally.
However, there are still three problems in this
experiment: 1) Because the accuracy of Optitrack is not
particularly high, there is an error in the collision of
virtual world objects. 2) Chroma key image synthesis is
just a superposition of two layers, and the real scene
cannot be fully integrated into the virtual scene. 3) Due to
the performance of the graphics workstation, the
synthesized video has a half-second delay. In the next
study, we will work hard to solve these problems.
10. References
[1] J. Arroyo-Palacios and R. Marks, "[POSTER]
Believable Virtual Characters for Mixed Reality," 2017
IEEE International Symposium on Mixed and Augmented
Reality (ISMAR-Adjunct), Nantes, 2017, pp. 121-123.
[2] H. Regenbrecht, K. Meng, A. Reepen, S. Beck and T.
Langlotz, "Mixed Voxel Reality: Presence and
Embodiment in Low Fidelity, Visually Coherent, Mixed
Reality Environments," 2017 IEEE International
Symposium on Mixed and Augmented Reality (ISMAR),
Nantes, 2017, pp. 90-99.
[3] Z. Zhang, B. Cao, D. Weng, Y. Liu, Y. Wang and H.
Huang, "Evaluation of Hand-Based Interaction for Near-
Field Mixed Reality with Optical See-Through Head-
Mounted Displays," 2018 IEEE Conference on Virtual
Reality and 3D User Interfaces (VR), Reutlingen, 2018,
pp. 739-740.

[4] Y. Baillot et al., "Evaluation of the ShapeTape tracker

for wearable, mobile interaction," IEEE Virtual Reality,
2003. Proceedings., Los Angeles, CA, USA, 2003, pp.
285-286.

[5] A. Drozina and T. Orehovacki, "Creating a tabletop

game prototype in unreal engine 4," 2018 41st
International Convention on Information and
Communication Technology, Electronics and
Microelectronics (MIPRO), Opatija, 2018, pp. 1568-1573.

[6] E. Hegarini, A. B. Mutiara, A. Suhendra, M. Iqbal and

B. A. Wardijono, "Similarity analysis of motion based on
motion capture technology," 2016 International
Conference on Informatics and Computing (ICIC),
Mataram, 2016, pp. 389-393.

[7] X. Chen, M. Wang and Q. Wu, "Research and

development of virtual reality game based on unreal
engine 4," 2017 4th International Conference on Systems
and Informatics (ICSAI), Hangzhou, 2017, pp. 1388-1393.

[8] R. Arai and K. Murakami, "Hierarchical human

motion recognition by using motion capture system," 2018
International Workshop on Advanced Image Technology
(IWAIT), Chiang Mai, 2018, pp. 1-4.

[9] X. Zhu and K. F. Li, "Real-Time Motion Capture: An

Overview," 2016 10th International Conference on
Complex, Intelligent, and Software Intensive Systems
(CISIS), Fukuoka, 2016, pp. 522-525.

[10] S. Kim, "Virtual studio system for augmented reality

& Chroma key processing," 13th International
Conference on Advanced Communication Technology
(ICACT2011), Seoul, 2011, pp. 762-765.

[11] T. Pfeiffer, M. E. Latoschik and I. Wachsmuth,

"Conversational Pointing Gestures for Virtual Reality
Interaction: Implications from an Empirical Study," 2008
IEEE Virtual Reality Conference, Reno, NE, 2008, pp.
281-282.

479