Professional Documents
Culture Documents
SAE Technical 2021-01-6000
SAE Technical 2021-01-6000
Citation: Ali, S., Hassan, O., and Aldahash, S., “Autonomous Gesture-Based Control Drone Design,” SAE Technical Paper 2021-01-6000,
2021, doi:10.4271/2021-01-6000.
Abstract
motion-controlled drone. The developed prototype is an inte-
T
he current drone market holds a lot of low-cost drones grated system that includes an off-the-shelf drone, accelerom-
that can be controlled either remotely or autonomously. eter and gyroscope unit, flex sensor, gloves, digital potentiom-
The command controlling signals can be deployed by eter, VR headset, and microcontroller. The flex sensor attached
using different methods and techniques to enhance the capabili- to the hand gloves allows a fully autonomous control of the
ties and missions of the drone. The microcontroller is usually drone using the movement of the hand. Hence, the gloves will
used as the drone brain. Motion control and virtual reality (VR) allow for a more dynamic control of the drone. Furthermore,
techniques allow the user new possibilities for a more dynamic the virtual headset is connected to the onboard camera to
control experience. The goal of the present work is to design, improve the field of view. Detailed integrations and the interface
develop, and deliver a working prototype of an enhanced VR are discussed including the challenges and constraints.
Keywords
UAV, Motion, Control, Autonomous vehicles, Virtual reality,
Flex sensors
1. Introduction
Therefore, the rotary-wing drone will be used for the present
C
urrently, drones have incredible influence and impact work. A rotary-wing drone has many setups and configura-
on our lifestyles as it has brought infinite possibilities. tions depending on how many motors are used [14, 15]. A
In the twenty-first century, drones have been used in drone is called a helicopter, tri-copter, quad-copter, hex-
various places with multiple functions and use. Drones are copter, or octo-copter based on the number of motors: one
currently used in transportation [1] and have been making rotary motor, three rotary motors, four rotary motors, six
their way in the security industry [2]; having drones can help rotary motors, or eight rotary motors, respectively. Because
reduce the number of technical personnel needed in a of its simplicity, accuracy, and ease of use, a quad-copter was
company [3], and drones can do many tasks as well as humans chosen for the present analysis [16]. However, systems, such
or even better [4, 5, 6, 7, 8]. Simply creating a drone is not a as UAVs, are difficult to model accurately because they exhibit
big accomplishment, but adding all the extra parts to make it nonlinearity and show variations with time. Therefore, the
unique and outstanding is definitely a point of interest in the control system must address the issues of uncertainty, nonlin-
present study. The main focus of the present work is to discuss earity, and complexity. The unpredictable behavior of a UAV
and explain the integration and interface of these parts requires a fast and reliable control system. Motion-based
together with off-the-shelf drones. control offers solutions without an accurate mathematical
There are two common drone models [9]: a fixed-wing model and enhances the flexibility and fast response for
drone and a rotary-wing drone. A fixed-wing drone is an problems of this nature [17].
Unmanned Aerial Vehicle (UAV) that has a motor at the end Motion control is the ability to understand and use
of the drone for thrust power and a fixed-wing to generate the human body motions such as of the hand and face to commu-
lift force in order for it to fly [10]. On the other hand, a rotary- nicate with and control a computer system or device without
wing drone consists of one or more rotary motors that can making direct physical contact [18]. Motion control offers a
induce the lift force [11]. The key advantage of the rotary-wing wide variety of control options, which is why many people are
drone is that it can be more stable when it is up in the air and attempting to integrate motion control into their everyday
provides more stability in capturing the images [12, 13]. lives. The flexibility that comes with using motion control for
Downloaded from SAE International by Shaaban Ali, Wednesday, September 01, 2021
machines like televisions or drones is highly considerable. sensors whereas Section 3 explains the system integration
Motion control is used in a lot of applications, like controlling including system microcontroller, flex sensor MPU6050 accel-
a TV and playing video games or controlling other electrical erometer and gyroscope sensors, and electrical connections
devices [19, 20, 21]. in detail. The Conclusion and future development are
In the literature, there are a few applications and discussed in Section 4.
researchers attempt such as aura drones where the drone is
semi-autonomously controlled by gloves instead of Remote
Control (RC) transmitters or phone apps. In these applica-
tions, selected hand fingers are used for each appropriate 2. Instrumentation and
position, and the hand is used for the pitch axis as in [22, 23].
The hand gesture-based control is one of the most powerful
Sensors
motion controls [24]. Additionally, there are different detec-
tion techniques including computer vision-based systems [25] In the current work, five instrumentation and sensor compo-
and wearable glove sensor-based systems [26]. In [27], a virtual nents are used: (1) MPU6050 is used to provide the orientation
reality (VR) technology is integrated with the motion capture of the drone; (2) The flex sensor is used to control the operation
system and implemented to a QAV 250 quadcopter. of the four motors; (3) The Arduino Nano receives the signal
From the above literature and studies, there is a lack of from the MPU650 and flex sensor, processes the sensor data,
autonomous control of drones in terms of motion and gesture and conveys the output data to the digital potentiometer’ (4)
control. The motion and gesture control will enhance and The digital potentiometer AD5206 sends the final control
broaden the drone maneuverability. Moreover, the ease and output signals to the drone main circuit; (5) The drone main
simplicity of the implementation by using hands and fingers circuit board controls the drone movements.
will definitely increase the functionality of the control system. The MPU6050 sensor includes many built-in sensors on
Recently, quite a few applications in the industry are using one chip including MEMS accelerometers, gyroscopes, and a
gestures and VR control in the drone industry. However, in temperature sensor. The accelerometers and gyroscopes are
the present study, a low-cost off-the-shelf drone is used and used to detect the position and orientation of the drone [9]
integrated with the proposed gesture control technique. This during the real-time flight. Therefore, it gives an accurate
is a plug-and-play solution where it can be applied to different analog conversion hardware including a 16 bit for each
drone models in the market. The main contribution of the channel. This module is the most important component as it
current study is to develop and integrate a low-cost gesture has the ability to capture the three axes’ (x, y, z) positions
control technique into off-the-shelf drones. simultaneously. In addition, it consists of an I2C bus that
In the present work, a glove with a flex sensor is used to interfaces with the Arduino board. The technical specifica-
control the drone. This type of control will provide more effec- tions of an MPU6050 are given in Table 1.
tive functionality than traditional control systems. Gesture A flex sensor, with the specifications given in Table 2,
control will introduce limitless motions that can be made is used as the main sensor in controlling the drone’s motors.
using the hand. This gives the ability to execute more When the flex sensor bends, the output voltage will accord-
commands. Hand gesture-based control also allows more ingly change. The output voltage consequently controls the
dynamic control over a drone using the physical movements drone motors; a high bend movement will result in
of the hand. These movements are used to make motion increasing the output voltage and hence will increase the
control more viable. The applications of using hand gesture- motor speed, and vice versa. This will allow the user to
based control for drones can be seen in various industrial accurately control the height of the drone without holding
projects. In 2017, Ntention were successfully able to test their an RC controller.
fabricated gloves for controlling a drone [28]. Also, the DJI The drone is controlled through the Arduino Nano v3.0
drone company introduced a drone that can be controlled by microcontroller board, which is a surface mount breadboard
using hand motion [29]. Recently, Goolsky company was able embedded version with an integrated USB. The microcon-
to make a drone that is controlled by a controller attached to troller used in the Arduino Nano is ATmega328; see Table 3
the hand, and when the hand moves, the drone will start for the technical specifications.
moving too.
In the present work, a hand gesture-based control will
be developed and integrated with an off-the-shelf drone. A
flex sensor attached to a hand glove will be used to detect the TABLE 1 Technical specifications of the MPU6050 unit.
hand and fingers’ movements. The integration, interface, and
development of the system will be described in detail. The Power supply 5V
system includes an off-the-shelf drone, accelerometer and Com. modes Standard I2C
gyroscope unit, flex sensor, gloves, digital potentiometer, VR protocol
headset, and microcontroller. The Drone will be entirely Built-in modules 16-bit AD converter,
controlled through a glove where selected fingers for each 16-bit data output
© SAE International.
required position are identified, and the hand is used for the Gyroscope range 250, 500, 1000,
2000 °/s
pitch axis.
The rest of the paper is organized as follows. Section 2 Accel. range 2, 4, 8, 16g
gives a detailed description of the instrumentations and Pin pitch 2.54 mm
Downloaded from SAE International by Shaaban Ali, Wednesday, September 01, 2021
© SAE International.
TABLE 3 Technical specifications of Arduino Nano v3.0
(Data is taken from Ref. [30]).
Operating 5V
voltage
Digital I/O pins 14 (6 pins for PWM
output)
Analog input 8 pins
pins FIGURE 3 The used digital potentiometer AD5206.
DC current per 40 mA
I/O pin
© SAE International.
3. System Integration
The main circuit board receives the signals from different
sensors. As shown in Figures 2 and 5, the original RC unit is
© SAE International.
FIGURE 4 Complete hardware configuration of the FIGURE 6 Digital potentiometer AD5206 with a PA0009
controller unit of the drone. adapter (potentiometer board).
© SAE International.
© SAE International.
FIGURE 5 Original RC unit of the drone.
© SAE International.
Figure 6 shows the digital potentiometer AD5206 along
with the PA0009 adapter. The potentiometer is used to send
the commands to the drone where a SOIC configuration is
deployed. The digital potentiometer is installed on the adapter FIGURE 8 Power supply to the main circuit with the voltage
PA0009 potentiometer board as shown in Figure 6. Then a regulator.
24-pin socket is used to facilitate integration and usage of the
potentiometer board with the main circuit board, as shown
in Figure 7.
The required supply voltage to power all the electronics
and sensors is 9 V DC. The electronic components on the
circuit board require a fixed current as well as voltage while
the power supply provides an unstable voltage output that may
result in inconsistent, inaccurate, and fault sensors data and
may also cause malfunction of some electronic components.
Therefore, the DC power supply is regulated by using a 7805
IC voltage regulator. The function of the voltage regulator is
to maintain the output voltage from the power supply at a
constant value. As shown in Figure 8, node 16 is connected
© SAE International.
5 V DC supply. the following subsections give more details FIGURE 9 Hand-movement drone control using a glove.
for each integrated component.
© SAE International.
code is included in the Appendix.
FIGURE 11 Connections of the MPU6050 sensor with the FIGURE 13 Assembly of the digital potentiometer and the
Arduino Nano. main transmitter circuit.
© SAE International.
© SAE International.
output voltage which is regulated from the battery 5V and the
second wire to the ground for both of MPU6050 and Arduino
Nano board.
The wiring connections of the Arduino and the digital
potentiometer are first made by connecting the two boards to
an empty breadboard and then using wires to connect the During the assembly process, various connections are
boards. Port D13 on the Arduino is connected to port CLK, made. As shown in Figure 13, Terminal B is connected to
which is the serial clock input on the digital potentiometer. channel 1, Wiper RDAC 4 is connected to channel 2, Terminal
Port D11 from the Arduino is connected to the SDI (Serial A RDAC 4 is connected to channel 3, and Wiper RDAC 1 is
Data Input). Port D10 from the Arduino is then connected to connected to channel 4. The main transmitter circuit is
the CSI (Chip Select Input). On the Digital Potentiometer, connected to a 4.5 voltage and is grounded. As for the digital
port Vdd is connected to the positive power supply. Finally, potentiometer, Terminal A RDAC 3, Terminal A RDAC 1,
the negative power supply port Vss and GND are connected Terminal A RDAC 2, and Terminal A RDAC 4 are all
to the ground, as shown in Figure 12. connected to a 4.5 voltage supply. On the other hand, Terminal
B RDAC 4, Terminal B RDAC 2, Terminal B RDAC 1, and
Terminal B RDAC 3 are connected to the ground.
4. Conclusion
potentiometer.
control is carried out for further enhancement and control of Unmanned Fixed-Wing Aerial Vehicle Based on Alternative
the UAV. Also the issue of delayed response will be addressed Energy,” International Journal of Aerospace Engineering 2019
in the off-the-shelf drone industry. This will provide a plug- (2019): 1-13.
and-play solution for various UAVs. 11. Liew, C.F., DeLatte, D., Takeishi, N., and Yairi, T., “Recent
Further work is currently conducted including the testing Developments in Aerial Robotics: A Survey and Prototypes
of the working prototype, acquiring performance data, and Overview,” Engineering, Computer Science, arXiv preprint
appropriate representation of the data to convey that the arXiv:1711.10085, (2017).
prototype control system works as expected. 12. Verbeke, J. and De Schutter, J., “Experimental
Maneuverability and Agility Quantification for Rotary
Unmanned Aerial Vehicle,” International Journal of Micro
Acknowledgment Air Vehicles 10 (2018): 3-11.
This work is supported by Sandooq Al Watan Funding scheme, 13. Tuzcu, I., Awni, K., and Gonzalez-Rocha, J., “Stability
UAE Government. Prediction of a UAV,” SAE Int. J. Aerosp. 4, no. 2 (2011): 1441-
1448, https://doi.org/10.4271/2011-01-2783.
14. Mairaj, A., Baba, A., and Javaid, A., “Application
Specific Drone Simulators: Recent Advances and
References Challenges,” Simulation Modelling Practice and Theory
94 (2019): 100-117.
1. Kellermann, R., Biehle, T., and Fischer, L., “Drones for Parcel 15. Burggräf, P., Pérez Martínez, A.R., Roth, H. et al.,
and Passenger Transportation: A Literature Review,” “Quadrotors in Factory Applications: Design and
Transportation Research Interdisciplinary Perspectives 4 Implementation of the Quadrotor’s P-PID Cascade Control
(2020): 100088. System,” SN Applied Science 1 (2019): 722.
2. Horváth and Partners, “Urban Air Mobility Study Report,” 16. Agrawal, K. and Shrivastav, P., “Multi-Rotors: A Revolution
Stuttgart, 2019. in Unmanned Aerial Vehicle,” International Journal of
3. Aurambout, J., Gkoumas, K., and Ciuffo, B., “Last Mile Science and Research (IJSR) 4, no. 11 (2015): 1800-1804.
Delivery by Drones: An Estimation of Viable Market 17. Hariprasad, V., Yaswanth, M.S., Sathish, V., Yamuna, N.
Potential and Access to Citizens across European Cities,” et al., “Autonomous Quadcopter for Building Construction
Europe Transportation Research Review 11, no. 1 (2019): 1-21. Monitoring,” SAE Technical Paper 2020-28-0515, 2020,
4. Ali, S., Al Dhaheri, M., Dawson, P., and Anavatti, S., https://doi.org/10.4271/2020-28-0515.
“Autonomous Water Sampling Payload Design,” 18. Sarker, A., Patel, K.A., Ram, R.K.G., and Capoor, G.K.,
International Review of Aerospace Engineering 13, no. 3 “Gesture Control of Drone Using a Motion Controller,” in
(2020): 120. International Conference on Industrial Informatics and
5. Amin, R., Li, A., and Shamshirband, S., “A Review of Computer Systems (CIICS), Sharjah, UAE, 2016.
Quadrotor UAV: Control Methodologies and Performance 19. Fu, Z.Q., Li, H., Jiang, Z.H., Ni, W.C. et al., “Design of a
Evaluation,” International Journal of Automation and Strategy for UAV to Avoid Obstacle and Its Verification in
Control 10, no. 2 (2016): 87-103. VR,” in IEEE 7th Annual International Conference on CYBER
6. Van Wynsberghe, A. and Comes, T., “Drones in Technology in Automation, Control, and Intelligent Systems
Humanitarian Contexts, Robot Ethics, and the Human- (CYBER), Honolulu, HI, 2017.
Robot Interaction,” Ethics Information Technology 22, no. 1 20. Fernando, H.C.T.E. et al., “Modelling Simulation and
(2020): 43-53. Implementation of a Quadrotor UAV,” in International
7. Fernandes de Oliveira, R., Schubert, F., and Potyka, V., Conference on Industrial and Information Systems IEEE,
“Flexible Trajectory Planning Framework Using Optimal Peradeniya, Sri Lanka, 2013.
Control for Rotary and Fixed-Wing Aircrafts Mission 21. Postal, G.R., Pavan, W., and Rieder, R., “A Virtual
Planning and Target-Pursuit,” SAE Int. J. Aerosp. 6, no. 2 Environment for Drone Pilot Training Using VR Devices,”
(2013): 703-713, https://doi.org/10.4271/2013-01-2264. in XVIII Symposium on Virtual and Augmented Reality
8. Carter, N., Hashemian, A., and Mckelvey, N., “An (SVR), Gramado, RS, Brazil, 2016.
Optimization of Small Unmanned Aerial System (sUAS) 22. Xia, X., Pun, C.-M., Zhang, D., Yang, Y. et al., “A 6-DOF
Image Based Scanning Techniques for Mapping Accident Telexistence Drone Controlled by a Head Mounted Display,”
Sites,” SAE Int. J. Adv. & Curr. Prac. in Mobility 1, no. 3 in IEEE Conference on Virtual Reality and 3D User Interfaces
(2019): 967-995, https://doi.org/10.4271/2019-01-0427. (VR), Atlanta, GA, 2019.
9. Cabreira, T.M., Brisolara, L.B., and Ferreira, P.R. Jr., “Survey 23. Duan, T., Punpongsanon, P., Jia, S., Iwai, D. et al., “Remote
on Coverage Path Planning with Unmanned Aerial Environment Exploration with Drone Agent and Haptic
Vehicles,” Drones 3, no. 1 (2019): 4. Force Feedback,” in IEEE International Conference on
10. Escobar-Ruiz, A.G., Lopez-Botello, O., Reyes-Osorio, L., Artificial Intelligence and Virtual Reality (AIVR), San Diego,
Zambrano-Robledo, P. et al., “Conceptual Design of an CA, USA, 2019.
Downloaded from SAE International by Shaaban Ali, Wednesday, September 01, 2021
24. Berezhnoy, V., Popov, D., Afanasyev, I., and Mavridis, N., 27. Anderson, N., Gao, J., Whitman, E., and Gururajan, S.,
“The Hand-Gesture-Based Control Interface with Wearable “Development of a Virtual Reality Environment (VRE) for
Glove System,” in 15th International Conference on Intuitive Drone Operations,” SAE Technical Paper 2017-01-
Informatics in Control, Automation and Robotics, Portugal 2070, 2017, https://doi.org/10.4271/2017-01-2070.
Porto, Vol 2, 2018. 28. Ntention Company Website, last accessed June 1, 2010,
25. Rautaray, S.S. and Agrawal, A., “Vision Based Hand Gesture https://ntention.com/posts/astronaut-smart-glove/.
Recognition for Human Computer Interaction: A Survey,” 29. “Spark User Manual,” last accessed May 20, 2020, https://
Artificial Intelligence Review 43, no. 1 (2015): 1-54. dl.djicdn.com/downloads/Spark/20171031/Spark%20
26. Luzhnica, G., Simon, J., Lex, E., and Pammer, V., “A User%20Manual%20V1.6.pdf.
Sliding Window Approach to Natural Hand Gesture 30. “Technical Specifications of Arduino Nano V3,” https://store.
Recognition Using a Custom Data Glove,” in IEEE arduino.cc/usa/arduino-nano.
Symposium on 3D User Interfaces (3DUI), Greenville, SC,
31. MPU-6050 Accelerometer + Gyro, https://playground.
USA, 2016.
arduino.cc/Main/MPU-6050/, accessed April 2, 2020.
© 2021 SAE International. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means,
electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE International.
Positions and opinions advanced in this work are those of the author(s) and not necessarily those of SAE International. Responsibility for the content of the work lies
solely with the author(s).
ISSN 0148-7191