Implementation of Gesture based Human machine interfaces on
Desktop computers using Arduino Uno and Python.
C.M.Mahajan, Abhishek Sharma, Shaikh Zaid, Shaunak Deshpande,
Shivangi Shardul, Abhinav Shambharkar.
Department of Engineering, Sciences and Humanities (DESH)
s
Abstract
The project aims to establish Gesture based
Human Machine interfaces or HMIs as a viable
alternative to traditional input devices like
keyboards. It uses an Arduino Uno, two
Ultrasonic sensors (HC-SR04)and python. The
HC-SR04 are used for calculating the distance
between the user’s hands and the computer, the
Uno is needed to use this value of distance and
convert it into instructions for python. Python then
converts these instructions to Keystrokes which in
turn control computer functionalities like
controlling volume, pause and play a video etc.
Keywords – Arduino , Hand gestures, Ultrasonic
Sensor, PyAutoGUI, Python
I. INTRODUCTION
Gestures are widely used in a variety of devices
like smartphones and tablets for control .Actions
like pinching, panning and swiping have become
intuitive for people for controlling a phone. On the
other hand, using hand gestures to control a
Laptop/Desktop PC is a field that is not explored
up to its full potential. The project aims to prove
that gesture-based controlling of Laptops/Desktop
PCs is a viable alternative to conventional devices
like a keyboard, mouse or Joystick. It uses two
ultrasonic sensors and an Arduino Uno to
calculate the distance between the user’s hands
and the laptop in real time and returns this value to
Python, which in turn takes decisions as to what
action to perform (like Volume control,
Play/Pause video etc.) using the PySerial and
PyAutoGUI library.
III. METHODOLOGY/EXPERIMENTAL
Serial communication was set between Arduino
and Python with a baud rate of 9600. In Arduino,
as calculating the distance between hand and
sensor was to be done multiple times, a function
was written which calculates it. The function was
called in the main loop to get the distance. If the
distance between both hands and both sensors is
between 40-50 cm, ‘Play/Pause’ was printed. If
one hand was at constant distance of 50cm and the
other was between 40 and
50cm from the sensor, forward(right hand constant
distance)/Rewind(left hand constant distance) was
printed. A
different methodology was used for volume
control. If hand was moving towards the ultrasonic
sensor, Vup [volume up] was printed, and if it was
moving away Vdown [volume down] was printed.
To interpret the statements that were printed,
Python’s PyAutoGUI library was used. First,
connection was established with Arduino through
the COM port. Then, in an infinite loop, Python
would read the incoming words that were printed
by Arduino and then convert them into
corresponding keystrokes to perform that action.
IV. RESUTLS AND DISCUSSIONS
At the end of the experiment, it was noted that
when the distance between both the hands of the
user and the two sensors was between 40-50 cm, if
a video file was playing it would get paused and
vice versa. If the left hand was at a constant
distance and the other hand was between 40-50
cm, the video would rewind, and if the right hand
was at a constant distance of 50 cm and the other
hand was between 40-50 cm, the video would
forward. To turn the volume up, one of the hands
was moved towards the sensor and to turn it down,
it was moved away from the sensor.
XI. LIMITATIONS
The project suffers from the following limitations:
1. Reproducibility:
The project requires complex Arduino and
Python commands that aren’t easy to
reproduce on other systems without having
access to source code of the program.
that are fundamental to computers(opening,
controlling applications) can be implemented
using Gesture based system, in a flexible and
scalable manner.
ACKNOWLEDGMENT
This major project would not have been possible
without the valuable assistance of many people to
whom we are indebted, in particular, our project
guide Shri.C.M.mahajan sir (HOD DESH
Department).We would also like to thank Our
College for providing us with the necessary
components for our project. Our thanks also goes
to all the teachers of DESH Department who
helped us in many difficult situations regarding
the project and provided with the necessary
advice. A special word of thanks is to our
classmates and our families for providing us the
moral support.

REFERENCES
2. Python:
Since Python is required for this project to
 Arduino Official
work, any Laptop/Desktop that does not
support Python will not be able to run this
 MakeUseOf.com
project.
 MarkerSpaces.com

 YouTube
XII. FUTURE SCOPE
The project is proof of concept of a new way to
control computers. Gesture based Human Machine
Interfaces (HMIs) are common in Smartphones
and tablets but unheard of in the Laptop/Desktop
industry. This project can act as a steppingstone
to incentivize research in the industry for
development of Gesture based HMIs.

XIII. CONCLUSION.
It is evident from this project work that Gesture
based HMIs are a great alternative to traditional
systems (for e.g. keyboard, mouse) to control
laptops and desktop computers . Functionalities
VIII. FIGURES AND TABLES

Fig 1. Circuit Diagram

Fig 2. Working Model

 Fig 3. Arduino Uno Pin Diagram

Fig 4. HC-04 Ultrasonic Sensor Working