VR Digital Twin for a Smart Factory
Gicu-Călin Deac, Crina-Narcisa Georgescu, Costel-Emil Coteț
Abstract—This paper describes some author’s research in
developing collaborative virtual reality applications as an
interface for monitoring big data by creating a digital twin of
the factory and sync the movement of virtual machines with the
real ones. The platform allows an interactive reading of the
sensor telemetry and processes data, maintenance information
and access to a large technical library. For data acquisition and
reports, a novel image data method was used. The data values
that are encoded as pixel colors of images, using different
encoding methods for each data type (text, integer, float,
Boolean) are also encrypted using an image as a symmetric
encryption key and are stored in the cloud in a time base folder
structure, assuring a better data compression, security and
speed, compared with the existing solutions based on JSON and
NoSQL databases.
Index Terms — virtual reality, smart factory, digital twin, big
data, Industry 4.0
I. INTRODUCTION
In the implementation of a control system, according to I.
Gonzales et al. the key factor is the interconnection between
sensors, controllers, tools and cloud services, through a
secure communications network.
Monitoring, tracking and automation of technological
processes for both industrial and non-industrial settings
requires efficient transmission of information through
communications networks [1] Wired or wireless channels are
used to communicate data. Due to mobility and flexibility,
wireless communication has great advantages and will be
widely used in Internet-of-Things (IoT). Various wireless
protocols such as Wi-Fi, Bluetooth, ZigBee, 3G / 4G / 5G,
RFID, Z-Wave, IPV6 over personal area networks with low
power (6LoWPAN) and Near Field Communication (NFC)
are available [1].
Industrial Internet-of-Things (IIoT) is a term used to refer
to IoT applications in the industrial context and involves the
use of sensors and actuators, control systems, M2M
communications (from machine to machine), cloud storage
of Big Data, data analysis and security mechanisms [1].
According to L. Monostori et al. another challenging concept
emerges, starting with the Industry 4.0 paradigm:
Cyber-Physical Systems (CPS) which means that industrial
machines have sophisticated communication and smart
capabilities. Devices are linked over the network to detect,
monitor and act on physical components in the real world [2].
In order to simplify and optimize this large-scale integration,
proprietary protocols and new evolving communication
Manuscript received Febrary 20, 2020.
G . C. Deac PhD student University “Politehnica” of Bucharest (e-mail:
george.deac@impromedia.ro).
C. N. Georgescu PhD student University “Politehnica” of Bucharest
(e-mail: crina.deac@impromedia.ro).
C. E. Cotet PhD professor University “Politehnica” of Bucharest (e-mail:
costelemilcotet@gmail.com).
protocols must converge on a prevalent protocol platform [2].
One of the present existing protocols for industrial
communication is OPC-UA, a protocol that has been
standardized in the IEC 62541 series. OPC UA is an open and
secure platform that allows vendor-neutral programmable
logic controllers (PLCs) to communicate with each other and
up to the manufacturing level and into the production
planning or ERP system [3].
II. DATA ACQUISITION AND STORAGE
The present paper proposes a Client OPC UA application that
will read all the values from the OPC UA Server, on timed
cycles with a predefined frequency (each second for
example) and use a novel method for encoding and
encrypting of data and automatic uploading in the cloud.
The method consists of creating full-color images (16 million
colors) whose pixels are generated based on numerical or
alphanumeric values to be stored.
The color of each pixel can be defined by the
three-component R, G, and B with values between 0 and 255,
respectively 0 and FF in hexadecimal. The image size can be
defined according to the number of values to be archived and
their type.
For encoding, there are several methods, depending on the
type of values to be stored:
a. In the case of positive integers between 0 and 16777215,
the encoding can be done directly by converting the decimal
number to hexadecimal (ex. 16777215 becomes FFFFFF),
the six-digit hexadecimal string explodes into three groups of
two digits and each of this group is (R = 255, G = 255 and B =
255, in this case, a white pixel corresponding to the
maximum value that can be stored);
b. In the case of positive and negative integers, the same
encoding variant as above may be used by having the
maximum value to +/- 8388607, the numbers being encoded
by summing the storage value by 8388607 and decoding
being made by subtracting this value;
c. For float numbers they can be converted to 16-digit
hexadecimal values: 00.00.00.00.00.00.00.00, requiring three
pixels for storage (R1G1B1, R2G2B2, and R3G3, the last
component B3, is not used). Thus, any float value with any
number of decimals, positive or negative, (in IEEE 754
double-precision standard, the full range of numbers is
−9.999999×1096 through 9.999999×1096 [9]) can be
archived using a group of three consecutive pixels. The
image will have the number of pixels equals three times the
number of archived float values;
d. For String, alphanumeric values (UTF8), the conversion of
each alphanumeric character into hexadecimal is made
according to the UTF8 encoding table or a custom defined
table (for example A becomes hex: 41, z becomes hex: 7a),
so each pixel can store a hexadecimal to decimal number of 3
alphanumeric characters, one for each component R, G, B.
 Figure 1. Image generation
The image will have the number of pixels equal to one third
of the total number of characters.
e. Booleans values can be stored as alphanumeric values 0
and 1, on a pixel being stored three Booleans, one on each
component R, G and B.
For encrypting data of any kind, corresponding to the
above-mentioned encoding methods, an encryption image
with the number of pixels equal to 1/3 of the variables
number is used.
The automated generation of the image key is based on image
size and using random values for Rk, Gk, Bk. When variables
(properties values) are encoded, for each variable converted
in hexadecimal, the hexadecimal value returned from the
correspondent pixel in the image key will be added (for the
first variable will be used the first pixel, for the second
variable the second pixel and so on, or we can scramble the
pixel order based on an established rule). The hex average or
RGB values or only one hex value can be used (from R, G, or
B based on a variable ID established rule). If the resulting
value of R, G or B is greater than 255, (FF in hex) the result
will be decreased by 255 (FF in hex).
The decryption and decoding process is analogous but in
reverse. In other words, from the hexadecimal value returned
by decoding the RGB values of each group of pixels (based
on variable type and encoding method) the hexadecimal
value returned by RGB of the correspondent pixel in the
Image Key will be subtracted based on the established rule
and then based on the encoding method the variable value
will be decoded. If the value obtained by subtraction is
negative, this value will be increased with 255 (FF in hex)
before decoding the variable value. (Figure 1).
Some descriptive information may also be entered in the first
pixels of the storage image such as: time stamp, device ID,
the number of variables of each type, the size of varchar
variables (number of characters), like a dictionary, in this
way we can implement a more flexible solution. For
example: first 3 pixels will store the timestamp (as a float),
the fourth pixel will store the device ID (as integer), pixels 5,
6, 7, 8 will store data types, the number of variables of each
data type and the order of placement in the image (on each
pixel we will encode in each component: R: type of variable,
G: number of variables, B: placement order), pixel 9 will
store the number of characters of the varchar variable. For the
type of variable, we can assume values: 1 for Boolean, 2 for
integer, 3 for float, 4 for varchar. Regarding the order of
placement, we will sort the data by variable type before
encoding, we will count the number of variables of each type
and then we will choose an order of encoding for each type of
variables. The generated image will have in this case a
dictionary composed by 4 pixels (for timestamp) and 5 pixels
for variable dictionary.
In IoT applications, the OPC UA Client application will
generate the images based on the device properties values,
with a predefined frequency (example: 1 time/second) and
send them to the cloud using a socket, or any other protocol.
The PNG images names can be constructed on the following
model:
deviceUniqueID_timestampInMiliseconds.png
example: kep35_31494270124240.png
and will be stored on the cloud in an arborescent file structure
based on a model like:
/deviceUniqueID/t1/t2/t3/t4/t5/t6/t7/deviceUniqueID_timest
ampInMiliseconds.png
example:
/kep35/1/4/9/4/2/7/0/ kep35_1494270124240.png
The folder structure can be generated based on timestamp by
exploding the first 7 characters and store the images inside
the last subfolder (t7). This storing model allows a good
management of time-series data, to read, copy and replicate
only the desired intervals. This model also allows real-time
interrogations of device state at a specific time.
The PNG nondestructive compression, based on LZW offers
a great compression rate, comparing to JSON files, the same
amount of data being much smaller (about 30 times smaller).
III. VR INTERFACE
To develop the VR server-client multi-user platform as an
immersive interface we selected the 8agora engine
(https://8agora.com) that allows development by exposing a
large API through JavaScript and QML. The 3D environment
can be created by importing 3D models with PBR (physically
based rendering) materials in FBX open format. The 3D
objects can have all the textures embedded, or externally
linked and could include animations which are easily
triggered by scripting. The frame rate of the animations can
be changed accordingly with the speed of real machines.
8agora engine provide live audio communication and include
a server-side audio mixer that compute the attenuation based
on the distance between user’s avatars and spatial surround
panning based on angular position of the user user avatar,
providing a very realistic and immersive communication
between users. The engine also allows the use of
customizable 3D avatars and includes a powerful physics
simulation system for avatar movement and object
interaction. Based on HTC Vive or Oculus Rift trackers, or
by using a Leap Motion controller, all the joints motions of
the user are reflected in real time to the avatar and are also
broadcasted to the other users, using an avatar mixer
server-side process. In this way all the user movements are
shown in real time by all the participants of the simulation.
To enhance the nonverbal communication and body
language, the engine includes an advanced automated
animation system and a gesture interface for the users which
are using the platform without HMD and hand controllers, in
desktop mode. With a simple handshake, users can exchange
contact information, all contacts being displayed in a contact
list application. Using this contact list application, all your
contacts and friends can be displayed with their status, online
or offline and by a simple click on their names you can see
their location and have the option to follow them.
8agora engine allows also the access of web content inside a
web entity, which is a chromium-based web browser. Using
this option, to display web content in 3D environment, can be
developed a wide array of applications to extend the
functionality of the platform.
Figure 2 Outside view of the factory
This paper presents an application for a Smart Factory
(Fig.2), where Virtual Reality is used to create a digital tween
of a factory, where all the machines and processes are
simulated. Using the telemetry data and other information
accumulated in cloud from the factory, using the data image
method, the virtual digital tween can be synchronized, each
machine movement being in sync with the real ones. (Fig. 3)
Figure 3 Assembly line on the factory
Also, we have included a technical library and all the users
have access in real time to technical information regarding
the machines and processes. The virtual warehouse is
populated with spare parts and objects, based on a real time
synchronization with the ERP. Using this technology for
visualization, provide a better and immersive interface to all
data that can be consulted in real time with the team in a
collaborative way. By clicking on the components of the
machines the users can show the telemetry data from sensors,
information regarding the maintenance (can be implemented
also a predictive maintenance cloud application to see the
remaining working hours for each component), they can
access technical catalogs and brochures, working instructions
and more. The platform can be used also for training on
 Figure 4 Assembly line
machines, different scenarios for working conditions, hazard
and simulations being easy to be implemented using the
physics engine in combination with the fbx baked animations
triggered by scripting. In this way the new employees can be
trained to proper use the machines without stopping the
production process and affecting expensive machine. After
completing the training in VR they can easily use the
knowledge to operate the real machines and processes.
Figure 5. Virtual production line
IV. CONCLUSION
As a result of applications implemented in this paper, we
can conclude that Virtual Reality is an emerging technology
that can be used with success for visualization of industrial
processes. Future research will be done in order to implement
commands from the VR application to Factory processes.
This work has been funded by the European Social Fund from the
Sectoral Operational Programme Human Capital 2014-2020, through the
Financial Agreement with the title "Scholarships for entrepreneurial
education among doctoral students and postdoctoral researchers (Be
Antreprenor!)", Contract no. 51680/09.07.2019 - SMIS code: 124539.
REFERENCES
[1] I. Gonzales, A. J. Calderon, A. J. Barragan and J. M. Andujar,
Integration of Sensors, Controllers and Instruments Using a Novel OPC
Architecture Published online 2017
[2] L. Monostori, B. Kadar, T. Bauernhansl, S. Kondoh, S. Kumara, G.
Reinhart, O. Sauer, G. Schuh, W. Sihn, K. Ueda, Cyber-physical systems in
manufacturing. CIRP Ann-Manuf. Technol. 2016 [1]. Hype Cycle for
Emerging Technologies, Gartner Research, August 2008,
http://www.gartner.com/it/page.jsp?id=739613
[3] Podnar Žarko I., Broering A., Soursos S., Serrano M. Interoperability and
Open-Source Solutions for the Internet of Things. InterOSS-IoT 2016.
Lecture Not
[4]. https://8agora.com
[5] Burdea G., Coiffet Ph., La realite Virtuelle, Ed. Hermes, Paris, 1995
[6] Chaillou C., Froumentin M., La Synthese d’images, 1997, Ėcole
Universitaire d’ingenieurs de Lille, France
[7] Gicu Calin Deac, Crina Narcisa Georgescu, Cicerone Laurentiu Popa,
C.E. Cotet, Virtual Reality Exhibition Platform, Proceedings of the 29th
International DAAAM Symposium 2018, DOI:
10.2507/29th.daaam.proceedings.033
[8.] Gicu Calin Deac, Crina Narcisa Deac, C.E. Cotet, M. Ghinea,
TIMV-plus, a new approach of the immersive e-learning platforms, The 4th
International Conference on Economic Sciences and Business
Administration, 2017, DOI: 10.26458/v4.i1.12
Gicu-Calin Deac. Born in Baia Mare in 1970, PhD
student at University “Politehnica” of Bucharest,
master’s degree in Training Techniques in the Virtual
Environment at University “Politehnica” of Bucharest
2016, dipl. Engineer, University of North Baia Mare
(1995). He is entrepreneur and are working at
Impro-Media SRL (https://impromedia.eu) as
cofounder and CTO. He has published some articles and books in VR, AR
and IIoT topics.
Crina-Narcisa Deac (Georgescu). Born in Baia Mare
in 1973, PhD student at University “Politehnica” of
Bucharest, master’s degree in Training Techniques in
the Virtual Environment at University “Politehnica” of
Bucharest 2016, matematician, University of North
Baia Mare (1997). She is entrepreneur and are working
at Impro-Media SRL (https://impromedia.eu) as
cofounder and CEO. She has published some articles and books in VR, AR
and predictive maintenance topics.