Intelligent System Concept of an IoT Cameras

Network Application for an Unmanned Aerial

Vehicle Control via a Graphical User Interface
2022 6th International Scientific Conference on Information, Control, and Communication Technologies (ICCT) | 978-1-6654-9367-3/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICCT56057.2022.9976710

Alexander Dubelschikov Tatyana Tsoy Yang Bai

Tikhonov Moscow Institute of Laboratory of Intelligent Robotics College of Information Science and
Electronics and Mathematics, Systems (LIRS), Institute of Information Engineering, Ritsumeikan University
HSE University Technologies and Intelligent Systems, Biwako-Kusatsu Campus
Moscow, Russia Kazan Federal University Kusatsu, Shiga, Japan
aadubelschikov@miem.hse.ru Kazan, Russia yangbai@fc.ritsumei.ac.jp
tt@it.kfu.ru

Mikhail Svinin Evgeni Magid

College of Information Science and Laboratory of Intelligent Robotics
Engineering, Ritsumeikan University Systems (LIRS), Institute of Information
Biwako-Kusatsu Campus Technologies and Intelligent Systems,
Kusatsu, Shiga, Japan Kazan Federal University
svinin@fc.ritsumei.ac.jp Kazan, Russia
HSE University
Moscow, Russia
magid@it.kfu.ru

Abstract — an important goal of modern robotics is to
benefit from a growing availability of internet of things (IoT)
devices. A single IoT device and their networks could
significantly improve user awareness and ease a control of
teleoperated robots. This paper discusses a concept of employing
stationary IoT cameras in order to improve the convenience of
an unmanned aerial vehicle (UAV) control by displaying
additional visual information directly to a UAV control panel of
an operator. The concept employs robot operating system (ROS)
and PX4-LIRS UAV model. The system will be initially
modelled in Gazebo simulator and then transferred onto real
hardware devices.
Keywords — IoT, PX4 LIRS_UAV, ESP32 CAM, OpenCV
I. INTRODUCTION
Unmanned aerial vehicles (UAV) based applications,
including quadcopters, are one of the most promising
directions in robotics today [1]. Used for civil needs, for
example, in smart agricultural farming [2], warehouse
management [3], traffic analysis [4], multi-agent coverage
of large-scale outdoor environments [5] or search and
rescue operations [6], [7], as well as for military purposes
[8], [9], UAVs are the subject of many studies, including
those aimed at improving various control criteria. Ease of
control is one of the most important criteria for a UAV
operator. According to statistics, from 21% to 67% of
UAV-related accidents occur due to a human factor [10].
Timely information, accurate, intuitive, responsive
control, and additional sources of information increase
operator awareness and a probability of complex tasks
success, and reduce a number of errors that could lead to a
UAV crash.
Researchers offer various ways of solving UAV
control convenience issues. In [11] authors proposed to
control a UAV using gestures that are tracked by several
cameras. Tactile interface in [12] adjusted UAV’s speed
depending on a pressure force, which was applied by an
operator to a control joystick. In [13] authors offered to
employ control barrier functions to generate force
feedback that helped a human operator to control a
quadrotor UAV in a safe manner. Moreover, haptic
feedback performed well in situations, where visual
interface suffered from delays or errors, which occurred
due to technical imperfection and interference [14].
Another approach to ease a control considers special
sub-systems, which simplify control and automatically
dodge obstacles. A visual odometry solution based on
RGB-D cameras for a short-term position estimation and
a local planner that generated trajectories and provided
obstacles’ avoidance was proposed in [15]. Simultaneous
localization and mapping (SLAM) algorithms could also
be employed in order to improve teleoperation safety for a
UAV, e.g., FastSLAM [16].
However, one of the most popular approaches
concentrates on visual interfaces. In [17] authors proposed
a visual servoing solution for a tethered UAV
teleoperation that uses a grid map in order to solve a
problem of a high video stream load, which prevents video
stutters and reduces a frame rate. In [18] authors discussed
two novel visualizations, which were designed to obtain
higher user acceptance ratings for a haptic collision
avoidance system by visualizing force fields. A concept of
using adaptive interfaces for a simultaneous control of
several UAVs by a single human operator was presented
in [19]. A solution for improving a control interface in [20]
suggested to employ an additional accompanying
quadcopter, whose camera images helped to control a
main UAV in a field of view of the additional quadcopter
while creating a third-person control effect.
This paper presents an approach of a UAV-based
environment exploration that uses a network of stationary
Internet of Things (IoT) cameras located in an indoor
environment. The exploration considers estimation of

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

Authorized licensed use limited to: University of East London. Downloaded on February 26,2023 at 21:21:37 UTC from IEEE Xplore. Restrictions apply.
potentially dangerous for a human variables and situations
in the environment, which could not be fully tracked solely
by the available indoor IoT cameras, e.g., within a typical
rescue operation [21]. In [22] authors estimated that a total
number of IoT cameras, which are expected to be placed
in the world by 2030, could reach 13 billion. These
cameras could be employed in our proposed approach in a
similar way to IoT camera based mobile robot indoor
localization [23]. The proposed approach targets to
improve situation awareness for an operator and ease the
UAV control while minimizing additional onboard
equipment load of the UAV.
II. SYSTEM DESIGN
The developed by Laboratory of Intelligent Robotic
Systems (LIRS) of Kazan Federal University UAV PX4-
LIRS [24] was chosen as a target UAV for the proposed
approach implementation (Fig. 1, top). At the first stage of
this work, a virtual model of the UAV (Fig. 1, bottom) in the
Gazebo simulator is used. The UAV is controlled with a set
of control commands that are used for the real UAV [25]. The
cameras that are placed in a virtual environment could
directly transfer data to a graphical user interface (GUI),
which allows an operator to control the UAV. The GUI
allows a manual switch between cameras in order to employ
the most suitable one for a current operational task support.
Fig. 1. PX4-LIRS UAV (top) and its virtual model for the Gazebo simulator
(bottom).
III. GRAPHICAL USER INTERFACE
GUI for the proposed system was developed in Qt and
rQt, which is its special version for ROS. It was divided into
two tabs: Control Window and Map Window.
Control Window (Fig.2) contains:
Authorized licensed use limited to: University of East London. Downloaded on February 26,2023 at 21:21:37 UTC from IEEE Xplore. Restrictions apply.
• UAV control buttons for 3D positioning and
orientation: throttle up/down, yaw right/left, pitch up/down,
roll left/right.
• Return - a button that initializes an autonomous
UAV return to its starting position. The autonomous return
could be triggered by an operator manually or starts
automatically if a connection between the UAV and the
operator is lost [26].
• A sliding controller and a corresponding input
window (which duplicate each other and could be selected by
the operator depending on his/her preferences) that allow to
set a UAV velocity.
• Four screens for displaying video data streams from
a UAV onboard camera (top central screen) and three
stationary IoT cameras (or virtual stationary cameras in a
simulation). The stationary cameras are labeled as Neighbor
1 (top left screen), Neighbor 2 (top right screen) and Indoor
(bottom central screen). While Indoor camera streams data
from a fixed IoT camera in the current location (a room, a
corridor, a hall or any other environment) of the UAV,
Neighbor cameras stream data from two other locations,
which are manually selected by the operator in Map Window
tab. Neighbor IoT camera allows the operator to see the UAV
from a third person perspective.
• Special indicators for a battery charge level, a wi-fi
connection and a compass.
• Takeoff and land buttons
The GUI corresponds to the PX4-LIRS UAV hardware
with a single onboard camera, which is directed downwards.
The first version of the system was implemented using real
IoT cameras, which are described in details in Section 4,
virtual stationary cameras in Gazebo and the Gazebo model
of PX4-LIRS UAV. Therefore the onboard camera streams
data from the Gazebo simulation, which is connected with
GUI using a standard rQt plugin. The real IoT cameras are
connected with GUI using a custom plugin, which employs
OpenCV library and connects to a local URL to get a mjpg
stream from the camera. The GUI control buttons are linked
to the topics of the UAV controller.
Fig. 2. Control Window.
Map Window (Fig.3) contains:
• Control cameras’ screens (top and bottom, on the
right) duplicate the corresponding screen of Control Window
to guarantee a continuous UAV tracking while the operator
sets other variables within Map Window.
• Map is a screen that displays a map of the entire
environment using RViz (top left). The RViz map allows
tracking a UAV position in the mission area and determining
adjacent rooms. The map could be available for the operator
in advance or constructed in real time using SLAM
algorithms [27]. The map is connected to the GUI with the
RViz plugin.
• Two context drop-down menus that allow a camera
selection for Neighbor 1 and Neighbor 2 screens of Control
Window. Each camera is labeled on the environment map so
that the operator could decide, which cameras to select in the
menus. In Fig.3 a particular example is demonstrated; the
cameras’ locations are displayed with red circles and are
simply labeled as “Indoor”, “Neighbor 1” and “Neighbor 2”.
The menus are integrated using a standard rQt camera image
output widget.
• Indicators for a battery charge level, a wi-fi
connection and a compass are duplicated in this window due
to their high importance in navigation tasks.
Fig. 3. Map Window.
IV. ESP32-CAM CAMERA BASED HARDWARE SETUP
A specialized module ESP-32 CAM [28] camera (Fig. 4)
is used as an indoor environmental camera of the system. This
module is a combination of an ESP-32 microcontroller with
a low power consumption, integrated Wi-fi and Bluetooth
controllers, and OV 2640 camera. This module was specially
created for solving issues with a streaming video
broadcasting. For the convenience of flashing this module, a
special UART converter ESP32-CAM-MB was used, which
simplified the programming process since other UART
converters require additional terminals in order to put the
ESP32-CAM into a programming mode. The outer case of
the camera was assembled using plastic containers and
powered by two 3.7V batteries of 2600 mAh capacity.
An image from the camera will be transferred via a web
server to a corresponding UAV GUI window. Data
throughput of camera-server and server-GUI channels are
important bottlenecks of the system and therefore it is critical
to follow the virtual world Gazebo simulation by a real world
experiments in various conditions.
Fig. 4. ESP32-CAM-MB (left), ESP 32-CAM (central), OV2640 (right).
Authorized licensed use limited to: University of East London. Downloaded on February 26,2023 at 21:21:37 UTC from IEEE Xplore. Restrictions apply.
Arduino IDE was used for the camera programing. Using
a modified standard sketch, we obtained a fairly clear image
from the camera, mainly working in a range 25-30 frames
(Fig.3, screens Neighbor 1,2 and Indoor). Due to a high load
and a high heat during the operation, we noticed rare cases of
frame rate drops and stutters. A possible solution to this
problem could be the use of more expensive microcontrollers,
including advanced models of Raspberry Pi. However, a
significant increase in the cost of IoT equipment is not worth
a slight improvement in productivity since scaling the
proposed approach to large indoor environments would
increase costs drastically and make it unfeasible.
Using a local IP address of the camera, we plan to transfer
the image from the camera directly to the UAV control panel
in the simulation using a special widget.
V. CONCLUSION
This paper proposed an approach of employing stationary
IoT cameras to improve an operator convenience of an
unmanned aerial vehicle (UAV) control by displaying
additional visual information directly to a UAV control panel
of an operator. The described system is based on PX4-LIRS
UAV and ESP 32-CAM environmental cameras, and
employs robot operating system (ROS). The system will be
initially modelled in Gazebo simulator and then transferred
onto real hardware devices. A potential bottlenecks, which
are related to data throughput channels are important
bottlenecks will be tested in real world experiments in various
conditions.
ACKNOWLEDGMENT
The second and fifth authors acknowledge the support of the
Kazan Federal University Strategic Academic Leadership
Program ("PRIORITY-2030").
REFERENCES
[1] N. Prosvirina, “Analysis and prospects for the development of
unmanned aircraft,” Moscow Economic Journal, pp. 560-575, 2021.
[2] P. Nintanavongsa and I. Pitimon, “Impact of sensor mobility on UAV-
based smart farm communications,” International Electrical
Engineering Congress (iEECON), pp. 1-4, 2017.
[3] A. Khazetdinov, A. Aleksandrov, A. Zakiev, E. Magid and K.-H. Hsia,
“RFID-based warehouse management system prototyping using a
heterogeneous team of robots,” International Conference on Climbing
and Walking Robots and Support Technologies for Mobile Machines
(CLAWAR), pp. 263-270, 2020.
[4] P. Peiro, C.Q. Gómez Muñoz and F. P. GarcíaMárquez, “Use of UAVs,
computer vision, and IoT for traffic analysis,” Internet of Things,
Springer, Cham, pp. 275-296, 2021.
[5] Y. Bai, Y. Wang, M. Svinin, E. Magid and R. Sun, “Adaptive multi-
agent coverage control with obstacle avoidance,” IEEE Control
Systems Letters, Vol. 6, pp. 944-949, 2022.
[6] S. Charalampidou, E. Lygouras, I. Dokas, A. Gasteratos and A.
Zacharopoulou, “A Sociotechnical Approach to UAV Safety for
Search and Rescue Missions,” International Conference on Unmanned
Aircraft Systems (ICUAS), pp. 1416-1424, 2020.
[7] E. Magid et al., “Artificial intelligence based framework for robotic
search and rescue operations conducted jointly by international teams,”
International Conference on Electromechanics and Robotics
“Zavalishin's Readings” Springer, Singapore, pp. 15-26, 2020.
[8] Z. Xiaoning, “Analysis of military application of UAV swarm
technology,” International Conference on Unmanned Systems (ICUS),
pp. 1200-1204, 2020.
[9] K. W. Williams, “A Summary of Unmanned Aircraft Accident-
Incident Data: Human Factors Implications, ” FAA The Defence
Technical Information Center Tech. Rep., 2004.
[10] K. Hartmann and K. Giles, “UAV exploitation: A new domain for
cyber power,” International Conference on Cyber Conflict (CyCon),
pp. 205-221, 2016.
[11] J. Rodríguez-Gómez, R. Tapia, A. Eguíluz, J. Martínez-de Dios and A.
Ollero, “UAV human teleoperation using event-based and frame-based
cameras,” Aerial Robotic Systems Physically Interacting with the
Environment (AIRPHARO), pp. 1-5, 2021.
[12] A. Kanso, I. H. Elhajj, E. Shammas and D. Asmar, “Enhanced
teleoperation of UAVs with haptic feedback,” IEEE International
Conference on Advanced Intelligent Mechatronics (AIM), pp. 305-
310, 2015.
[13] D. Zhang, G. Yang and R. P. Khurshid, “Haptic Teleoperation of
UAVs Through Control Barrier Functions (IEEE Transactions on
Haptics), ” IEEE Transactions on Haptics, pp. 109-115, 2020.
[14] T. M. Lam, V. D'Amelio, M. Mulder and M. M. van Paassen, “UAV
Tele-operation using Haptics with a Degraded Visual Interface, ” IEEE
International Conference on Systems, Man and Cybernetics pp. 2440-
2445, 2006.
[15] F. J. Perez-Grau, R. Ragel, F. Caballero, A. Viguria and A. Ollero,
“Semi-autonomous teleoperation of UAVs in search and rescue
scenarios,” International Conference on Unmanned Aircraft Systems
(ICUAS), pp. 1066-1074, 2017.
[16] J. Mendes, R. Ventura, “Safe teleoperation of a quadrotor using
FastSLAM, IEEE International Symposium on Safety, ” Security, pp.
1-6, 2012.
[17] M. Riestock, F. Engelhardt, S. Zug and N. Hochgeschwender, “User
study on remotely controlled UAVs with focus on interfaces and data
link quality,” IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pp. 3394-3400, 2017.
[18] V. Ho, C. Borst, M. Van Paassen and M. Mulder, “Increasing
Acceptance of Haptic Feedback in UAV Teleoperation by Visualizing
Force Fields,” IEEE International Conference on Systems, Man, and
Cybernetics (SMC) pp. 3027-3032, 2018.
[19] J. Cooper and M. A. Goodrich, “Towards combining UAV and sensor
operator roles in UAV-enabled visual search,” ACM/IEEE
Authorized licensed use limited to: University of East London. Downloaded on February 26,2023 at 21:21:37 UTC from IEEE Xplore. Restrictions apply.
International Conference on Human-Robot Interaction (HRI), pp. 351-
358, 2008.
[20] X. Xiao, J. Dufek and R. Murphy, “Visual servoing for teleoperation
using a tethered UAV,” IEEE International Symposium on Safety,
Security and Rescue Robotics (SSRR), pp. 147-152, 2017.
[21] N. Simakov, R. Lavrenov, A. Zakiev, R. Safin and E.A. Martínez-
García, “Modeling USAR maps for the collection of information on the
state of the environment,” International Conference on Developments
in eSystems Engineering (DeSE), pp. 918-923, 2019.
[22] A. Mohan, K. Gauen, Y. Lu, W. W. Li and X. Chen, “Internet of video
things in 2030: A world with many cameras,” IEEE International
Symposium on Circuits and Systems (ISCAS) pp. 1-4, 2017.
[23] K. Kononov, R. Lavrenov, T. Tsoy, E. A. Martinez-Garcia and E.
Magid, “Virtual experiments on mobile robot localization with an
external smart RGB-D camera using ROS,” IEEE Eurasia Conference
on IOT, Communication and Engineering (IEEE ECICE), pp. 656-659,
2021.
[24] A. Khazetdinov, A. Zakiev, T. Tsoy, M. Svinin and E. Magid,
“Embedded ArUco: a novel approach for high precision UAV
landing,” The 15th Siberian Conference on Control and
Communications, 2021.
[25] S. Mghabghab, I. H. Elhajj and D. Asmar, “Adaptive gain tuning for
teleoperation of quadrotors,” 2016 IEEE International
Multidisciplinary Conference on Engineering Technology (IMCET),
pp. 173-178, 2016.
[26] D. Bereznikov and A. Zakiev, “Network failure detection and
autonomous return for PMB-2 mobile robot,” International Conference
on Artificial Life and Robotics, pp. 444-447, 2020.
[27] E. Mingachev et al., “Comparison of ROS-based monocular visual
SLAM methods: DSO, LDSO, ORB-SLAM2 & DynaSLAM,”
International Conference on Interactive Collaborative Robotics
Springer, Cham, pp. 222-233, 2020.
[28] A. Kaur, A. Jadli, A. Sadhu, S. Goyal and A. Mehra, “Cloud Based
Surveillance using ESP32 CAM,” International Conference on
Intelligent Technology, System and Service for Internet of Everything
(ITSS-IoE) pp. 1-5, 2021.