VISION-BASED GESTURE -

CONTROLLED CURSOR NAVIGATION

SYSTEM
Nehal Ahmed A Mohammed Hashim A
Student, Department of Computer Science Student, Department of Computer Science
and Engineering, and Engineering,
Sathyabama Institute of Science and Sathyabama Institute of Science and
Technology, Technology,
Chennai, India Chennai, India
iamnihalaaa@gmail.com D Sudha hashim2462@gmail.com
Assistant Professor, Department of
Computer Science and Engineering,
Sathyabama Institute of Science and
Technology,
Chennai, India
sudha.dandapani.cse@sathyabama.ac.in

Abstract—Researchers all throughout the world are
working to improve the responsiveness of our gadgets
and to make it possible for them to operate with the
least amount of physical interaction. There are three
primary varieties of mice—wired, wireless, and
Bluetooth—that need electricity to function. Mice are
an essential tool in human-computer interaction. This
study presents a novel computer system that can be
used interactively without a real mouse. The device
recognizes hand motions without any additional
hardware thanks to advanced machine learning and
computer vision algorithms. This approach, which is
compatible with CNN models provided by mediapipe,
has tremendous potential for a variety of users,
especially those who struggle with conventional mouse
operation due to partial paralysis. The technology uses
computer vision to let people use the mouse by showing
various finger counts. Various hand motions are
required for actions like clicking and dragging objects.
Additionally, this system uses a camera as an input
device and runs on a single computer. The proposed
solution, which shows the camera's output on the
screen for user adjustment, utilizes Python and
OpenCV.
Keywords- Mediapipe, Convolutional Neural
Network, Computer Vision, Hand Recognition, Partial
Paralysis, Human-Computer Interaction, Human-
Robot Interaction.
I. INTRODUCTION
Within the field of computer vision, gesture recognition,
which focuses on recognizing hand movements as a kind of
visual communication has long been an intriguing
challenge. These hand gestures involve motions across the
middle of the palm, finger placement, and hand form. Hand
gestures can be categorized as either static or dynamic.
Static gestures consist of a single, steady hand form,
whereas dynamic gestures include a series of hand motions,
such as waving.
Distinguishing between gesture and posture, gesture focuses
on hand movement, whilst posture highlights the shape of
the hand. The mouse is a key instrument for Human-
Computer Interaction (HCI), however in some real-world
situations, such Human-Robot Interaction (HRI), it might not
be practical. Numerous studies have looked at alternative
HCI methods, and hand gestures have come to light as a
natural and intuitive replacement for the conventional
mouse, especially useful in situations such as human-
machine interfaces. While many people take technology for
granted, utilizing standard gadgets can be challenging for
those who have physical disabilities. Extreme impairments in
mobility can severely impair fine motor function, making it
difficult to interact with a regular mouse. Sufficient assistive
technology are essential for ensuring accessibility for those
who require it. The idea is to design a virtual mouse system
that replaces touch displays with a camera enabling easier-
to-use connection with the device. Using vision-based
Computer Vision (CC) can help a webcam reach its full
potential. Using specified labels, this method effectively
records and predicts hand motions. The ultimate objective is
to create a user-friendly virtual mouse system that
communicates with devices via webcam-based vision,
providing a good replacement for touch displays and
improving accessibility, particularly for those with physical
disabilities.
II. LITERATURE REVIEW
Researchers and automobile organizations have spent the
previous five years carefully examining the proposed
problem in an effort to come up with a solution. The
suggested remedies take multiple tacks, focusing on the
examination of various patterns.
In order to enhance user-computer interaction, Aashni Hariaa
et al. [1] presented "Hand Gesture Recognition for Human
Computer Interaction," which makes use of a backdrop
extraction and contours recognition algorithm. A variety of
sensor modalities, such as gestures, voice, facial expressions,
and body language, can be used to facilitate human contact.
Horatiu-Stefan et al. presented a paper titled "Human Hand
Gesture Based System for Mouse Cursor Control." This
model makes use of a computer, a webcam, and a blue hand traditional techniques such as manually entering buttons or
pad. Hand pad postures are used to guide operations. The adjusting the placements of a physical computer mouse.
OpenCV library and Visual C++ 2008 were utilized in the Instead of using a real computer mouse to do similar tasks, it
development of the model. uses computer vision and a camera to monitor and control
A "Mouse Control Using a Web Camera Based on Color different mouse movements.
Detection" was proposed by Abhik Banerjee et al. [2] to Firstly, the hand and its key characteristics are identified by
control click events and cursor movements by sensing the means of Mediapipe. Mediapipe detects a total of 21
color of the image. distinct colors correspond to distinct important points. To recognize the hand, the process uses
cursor controls, and clicking operations are carried out by Mediapipe's palm detection model and hand landmark
simultaneously recognizing these colors. This methodology model. When the hand landmarks are taken into
was developed with the help of MATLAB software and the consideration, palm detection is a simple process.
MATLAB Image Processing Toolbox.
Abhilash et al. [3] introduced the "Virtual Mouse Using
Hand Gesture" approach for manipulating the mouse
pointer and clicking actions. In this example, a red object
was used to build a mask. This approach was implemented
using Python software, which included a number of
modules and functions.
To deal with the mouse pointer and click actions, Alisha
Pradhana et al. [4] proposed the "Design of Intangible
Interface for Mouseless Computer Handling Using Hand
Gestures". The Microsoft Visual Studio integrated
development environment was used to apply this strategy,
which uses a convex hull algorithm. To minimize noise and
identify the user's hand, a red glove is utilized.
"A Real Time Hand Gesture Recognition System Using
Motion History Image" was proposed by Chiung Hsieh et
al. [5] to manipulate the mouse cursor. To minimize the
number of incorrect classifications, the recommended
method makes use of an adaptive skin color detection
model. A C++ software platform with the OpenCV image
processing package loaded was used to construct these
techniques.
"Cursor Control Using Hand Gestures" was the idea put out
by Pooja Kumari et al. [6] to operate a mouse using hand
motions that are recorded by a camera. Using this method,
the camera functions as a sensor to record and detect color
tips applied on the hand. Known as the marker-based
technique, it requires users to hold color-tip mouse cursors Figure 1 Co-ordinates or landmark in hand
in their hands. The MATLAB environment, the MATLAB
1. CAMERA
Image Processing Toolbox, and the OpenCV library were
all utilized in the implementation of this technique. The webcam that is connected to the computer or the
laptop's built-in camera provides the frames that the
system uses to operate. In order to accomplish this, a
III. MATERIALS AND METHODS
video capture object is set up, allowing for real-time
The program relies on specific software requirements
webcam video capturing. Frames are gradually
crucial for its proper operation. These essential
captured by the camera and then sent to the system
prerequisites ensure the functionality, performance, and
for additional processing.
usability of the project:
2. CAPTURING
 Python (version 3.6 – 3.8.5)
Through the use of an infinite loop, the camera
 pycaw captures every frame until the application ends.
 PyAutoGUI Following their extraction from the real-time video,
 AutoPy the frames undergo processing that changes their
 MediaPipe color space from BGR to HSV.
 NumPy
3. COLOR DETECTION AND MASKING
The project development process is based on these Color detection works by identifying color pixels in
predefined software requirements, which ensures the final the webcam frames that correspond to fingertips that
product satisfies anticipated functional, performance, and have colored caps on them. This is the first and
usability criteria. It is essential to remember that, depending crucial stage in the suggested system. This process
on the project's particular objectives and use cases, more yields a grayscale image in which the color cap is
extensive and specialized requirements can be necessitated. highlighted by pixel intensity, setting it out from the
Using a real-time camera to control cursor movement, this surrounding frame. The color cap itself and the
project proposes an innovative approach to Human- rectangle enclosing it are then monitored. Then, by
Computer Interaction (HCI) that offers an alternative to keeping an eye on these color caps, gesture detection
 becomes simpler.

4. GESTURE RECOGNITION
4.1 Mouse Movements
First, by extracting the coordinates of the centers of
the identified rectangles, the computation may be
performed to find the center of the two color objects
that were successfully detected. The built-in
OpenCV function is used to draw a line between
these two locations. Next, the following equation is
used to find the line's midpoint:

M=(Xa+Xb/2, Ya+Yb/2)

The mouse cursor tracks to the identified midway, it

then aligns itself with this midway. In this
configuration, the coordinates that are obtained from
the resolution of the camera-captured frames are Figure 2 Snapshot of Neutral Hand Gesture
transformed to match the resolution of the screen.
The mouse is assigned a preset location, and the
moment the mouse cursor reaches this spot, the
mouse starts functioning. This initiation point is
referred to as the "open gesture," enabling users to
control the mouse pointer effectively.

4.2 Mouse Clicking

In order to execute clicking actions, the suggested
method uses near movements. A new bounding box
forms at the boundary of the tracking bounding
boxes when the bounding boxes of rectangles get
closer to each other. When the newly formed
bounding box has grown to twenty percent of its
original size, the system clicks the left button.
Clicking on the bounding box executes this action.
A double click is made if the user holds this posture
for more than five seconds. The open motion is once
more used to indicate a click of the right button. One Figure 3 Snapshot of Neutral Click Gesture
finger is sufficient to click the appropriate button.
The fingertip is identified by the system by a color
cap, which prompts a right-button click.

4.3 Mouse Scrolling

With this approach, the user uses the open gesture
movement with three fingers, each colored cap, to
begin scrolling. The system performs a scrolling-
down motion when the user simultaneously lines up
three fingers and moves their location downward.

IV. RESULT AND DISCUSSION

In order to contribute to the evolving field of vision-based

machine-human interactions, we have utilized the
principles of computer vision and human-computer
interface (HCI) in our work. Our proposed work is mainly
concerned with hand gesture control of mouse
Figure 4 Snapshot of Mouse Cursor Movement
functionalities. The main features include the ability to
move the mouse, click with the left or right button, double
click, and scroll up or down.
 [6] Pooja Kumari ,Ghaziabad Saurabh Singh, Ghaziabad
Vinay and Kr. Pasi “Cursor Control using Hand Gestures”
International Journal of Computer Applications (0975 –
8887),2016.

[7] Haria, A., Subramanian, A., Asokkumar, N.,

Poddar, S., Nayak, J.S.: Hand gesture recognition for
human computer interaction. Procedia computer
science 115, 367– 374 (2017)
[8] Islam, M.Z., Hossain, M.S., ul Islam, R.,
Andersson, K.: Static hand gesture recognition using
convolutional neural network with data augmentation.
Figure 5 Snapshot of Click Gesture
In: 2019 Joint 8th International Conference on
Informatics, Electronics & Vision (ICIEV) and 2019
V. CONCLUSION
Using a camera, the Virtual Gesture Control Mouse is a
3rd International Conference on Imaging, Vision &
system that can move the mouse pointer and carry out Pattern Recognition (icIVPR). pp. 324–329. IEEE
operations in real time. Mouse navigation, icon selection, (2019)
and related operations like left, right, double click, and
scrolling have all been implemented. To track mouse cursor
[9] Jagannathan, M., Surya, M., BT, A.M.,
movements and icon selection, this system uses motion Poovaraghavan, R.: Finger recognition and gesture
detection and image comparison technologies. Our research based augmented keyboard (2018)
suggests that the system may be made more versatile by
[10] Keil, A., Albuquerque, G., Berger, K., Magnor,
adjusting the algorithms to other areas using a high-quality
camera and enough light. In the future, we want to improve M.A.: Real-time gaze tracking with a consumer-grade
the system by adding additional capabilities including video camera (2010)
interacting with multiple windows, reducing and resizing
windows, and shutting windows. The palm and many
fingers would be used in motion to carry out these
functions. This expansion is envisioned to provide a more
comprehensive and systematized user experience.

REFERENCES
[1] Aashni Hariaa , Archanasri Subramaniana ,
Nivedhitha Asokkumara , Shristi Poddara and Jyothi S
Nayak “Hand Gesture Recognition for Human Computer
Interaction” Proc.ICACC(International Conference on
Advances in Computing & Communications), 2017
August,pp367–374.
[2] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni
Bharadwaj,” Mouse Control using a Web Camera based
on Color Detection”,IJCTT,vol.9, Mar 2019.
[3] Abhilash S , Lisho Thomas, Naveen Wilson, and
Chaithanya “VIRTUAL MOUSE USING HAND
GESTURE” Proc. International Research Journal of
Engineering and Technology (IRJET), e-ISSN: 2395-0056
p-ISSN: 2395-0072,Apr-2018, Volume: 05 Issue:04.
[4] Alisha Pradhana , B.B.V.L. Deepak “Design of
Intangible Interface for Mouseless Computer Handling
using Hand Gestures” ICCCV(International Conference
on Communication, Computing and Virtualization), 2016,
oi: 10.1016/j.procs.2016.03.037.
[5] Chiung Hsieh, D.-Hua Liou, & D. Lee, , “A real time
hand gesture recognition system using motion history
image,” Proc. IEEE Int'l Conf. Signal Processing Systems
(ICSPS), 2.10.1109/ICSPS.2010.5555462, 2020.