A PROJECT STAGE-I REPORT

ON

“Virtual Mouse Control Using Hand Class Gesture”

SUBMITTED TOWARDS THE PARTIAL FULFILLMENT
OF THE REQUIREMENTS OF DEGREE OF

BACHELOR OF ENGINEERING
IN
ELECTRONICS AND TELECOMMUNICATION
BY
Mr. Your Name [Exam Seat No. 0000]
Gaurav Munjewar 71804123C
Geetesh G Kongre 71726431K

UNDER THE GUIDANCE OF

Prof. S. P. Deshmukh

DEPARTMENT OF ELECTRONICS AND TELECOMMUNICATION

STES’s NBN SINHGAD TECHNICAL INSTITUTES CAMPUS
NBN SINHGAD SCHOOL OF ENGINEERING
AMBEGAON (BK),
PUNE – 411041
2021-22
 NBN SINHGAD TECHNICAL INSTITUTES CAMPUS NBN
SINHGAD SCHOOL OF ENGINEERING, PUNE
DEPARTMENT OF ELECTRONICS AND
TELECOMMUNICATION ENGINEERING

CERTIFICATE
This is to certify that the Project Stage-I entitled

“Virtual Mouse Control Using Hand Class Gesture”

Submitted by,

Mr. Your Name [Exam Seat No. 0000]

Gaurav Munjewar 71804123C

Geetesh G Kongre 71726431K

towards the partial fulfillment of the degree of Bachelor of Engineering in Electronics and
Telecommunication as awarded by the Savitribai Phule Pune University, at NBN Sinhgad
School of Engineering.

Prof. Sunita Deshmukh Dr. Makarand . M. Jadhav Dr. Shivprasad P. Patil

Guide Head of Department Principa
Contents–
1. Introduction
2. Motivation
3. Scope & Objective
4. Literature
5. Problem Statement
6. Work Schedule
7. Required Hardware and Software
8. Advantages & Disadvantages
9. Conclusion
10. References
 Introduction
The most efficient and expressive way of human communication is through hand
gesture, which is a universally accepted language. It is pretty much expressive
such that the dumb and deaf people could understand it. In this work, real-time
hand gesture system is proposed. Experimental setup of the system uses fixed
position low cost web camera high definition recording feature mounted on the
top of monitor of computer or a fixed camera on a laptop, which captures
snapshot using Red Green Blue [RGB] color space from fixed distance. This work is
divided into four stages such as image preprocessing, region extraction, feature
extraction, feature matching. Recognition and the interpretation of sign language
is one of the major issues for the communication with dump and deaf people. In
this project an effective hand gesture segmentation technique has been proposed
based on the preprocessing, background subtraction and edge detection
techniques [1]. Pre-processing is defined as procedure of formulating data for
another process. The main objective of the preprocessing process is to transform
the data into a form that can be more effectively and effortlessly processed. In
the proposed work, the pre-processing techniques are created on the basis of
different types of combinations from the subsequent hand gesture image
processing operations such as capturing image, removing noise, background
subtraction, and edge detection and these image processing methods are
discussed.
Within the past few years, as computer technology continues to develop, people
want more compact electronic devices. Human Computing Interaction (HCI),
particularly gesture recognition and object recognition, is becoming increasingly
important. In our project, we introduce a method for controlling the mouse
system using a video device (Mouse tasks). In today's world, most cell phones
communicate with the user via touch screen technology. However, this
technology is still prohibitively expensive for use on desktops and laptop
computers. Generally, a gesture is a symbol for physical or emotional behaviour.
It consists of both body and hand gestures. Gestures can be used to communicate
between humans and computers. Human-computer interaction (HCI) began in the
early 1980s as a field of study and practice. The name "virtual mouse" conveys a
clear idea about our project. The virtual mouse establishes a virtual connection
between the user and the machine without the use of any hardware. This gesture
recognition system can capture and track the fingertips of a person wearing a
color cap with a webcam, and the system detects the hand's color and
movements and moves the cursor along with it.
Motivation
 The existing system consists of the generic mouse and trackpad system
of monitor controlling and the nonavailability of a hand gesture system.
The remote accessing of monitor screen using the hand gesture is
unavailable. Even-though it is largely trying to implement the scope is
simply restricted in the field of virtual mouse.

 The existing virtual mouse control system consists of the simple mouse
operations using the hand recognition system, where we could perform
the basic mouse operation like mouse pointer control, left click, right
click, drag etc. The further use of the hand recognition is not been made
use of. Even-though there are a number of systems which are used for
hand recognition, the system they made used is the static hand
recognition which is simply recognition of the shape made by hand and
by defining an action for each shape made, which is limited to a number
of defined actions and a large amount of confusion.

 A lot of virtual markers are available in market, but it can only function
as a marker alone. A high level processor is used to process the data and
used as a virtual marker, but it can do even more function like mouse
functions. Its capability is not being utilized fully to its ability to function
and hence the product becomes under-loaded and hence costlier in
terms of market price versus function.
Scope & Objective

Scope :-
 Using the current system even-though there are a number of quick
access methods available for the hand and mouse gesture for the
laptops, using our project we could make use of the laptop or web-cam
and by recognizing the hand gesture we could control mouse and
perform basic operations like mouse pointer controlling, select and
deselect using left click, and a quick access feature for file transfer
between the systems connected via network LAN cable. The project
done is a “Zero Cost” hand recognition system for laptops, which uses
simple algorithms to determine the hand, hand movements and by
assigning an action for each movement.
 But we have mainly concentrated on the mouse pointing and clicking
actions along with an action for the file transfer between connected
systems by hand action and the movements. The system we are
implementing which is been written in python code be much more
responsive and is easily implemented since python is a simple language
and is platform independent with a flexibility and is portable which is
desirable in creating a program which is focused in such an aim for
creating a Virtual Mouse and Hand Recognition system. The system be
much more extendable by defining actions for the hand movement for
doing a specific action. It could be further modified to any further extent
by implementing such actions for the set of hand gestures, the scope is
restricted by your imagination.
Objective:-
This Virtual Mouse Hand Recognition application uses a simple color cap on the
finger without the additional requirement of the hardware for the controlling of
the cursor using simple gestures and hand control. This is done using vision based
hand gesture recognition with inputs from a webcam.
Literature Survey
The current system is comprised of a generic mouse and trackpad monitor control
system, as well as the absence of a hand gesture control system. The use of a
hand gesture to access the monitor screen from a distance is not possible. Even
though it is primarily attempting to implement, the scope is simply limited in the
virtual mouse field.

The existing virtual mouse control system consists of simple mouse operations
using a hand recognition system, in which we can control the mouse pointer, left
click, right click, and drag, and so on. The use of hand recognition in the future will
not be used. Even though there are a variety of systems for hand recognition, the
system they used is static hand recognition, which is simply a recognition of the
shape made by the hand and the definition of action for each shape made, which
is limited to a few defined actions and causes a lot of confusion. As technology
advances, there are more and more alternatives to using a mouse.

The following are some of the techniques that were employed:

A) Eye Control: The cost of modern eye gaze systems is decreasing. These enable
users to move the pointer on the screen solely by moving their eyes. Instead of
mouse buttons, a dwell delay feature, blinks, or a switch are used. The Tobi PC
Eye Go is a peripheral eye tracker that lets you use your eyes to control your
computer as if you were using a mouse.
1. B) Touch Screens: Touch screens, which were once seen as a niche
technology used primarily in special education schools, have now become
mainstream. Following the success of smartphones and tablets, touch-
enabled Windows laptops and all-in-one desktops are becoming more
common. Although this is a welcome new technology, the widespread use
of touch screens has resulted in a new set of touch accessibility issues.
However, each of the methods below has its own set of disadvantages. The
use of the head or eyes to control the cursor regularly can be hazardous to
one's health. This can lead to a number of problems with health. When
using a touch screen, the user must maintain their focus on the screen at all
times, which can cause drowsiness. By comparing the following techniques,
we hope to create a new project that will not harm the user's health.
Problem Statement
• To design motion tracking mouse which detect finger movements gestures instead
of physical mouse.
• To design an application (.exe file) with user friendly user interface which
provides feature for accessing motion tracking mouse feature.
• The camera should detect all the motions of hand and performs the operation of
mouse.
• Implement such code where motion tracker mouse has drag & drop feature along
with scrolling feature.
• User Interface must be Simple & easy to understand.
• Physical mouse is subjected to mechanical wear and tear.
• Physical mouse requires special hardware and surface to operate.
• Physical mouse is not easily adaptable to different environments and its
performance varies depending on the environment.
• Mouse has limited functions even in present operational environments.
• All wired mouse and wireless mouse have its own lifespan.
• Implement such code where camera can recognize each and every finger
movement & responds according to it:
Methodology
Developed an AI-based approach for controlling the mouse movement using
Python and OpenCV with real-time camera that detects hand landmarks, tracks
gesture patterns instead of a physical mouse.
Proposed System
Using the current system even-though there are a number of quick access
methods available for the hand and mouse gesture for the laptops, using our
project we could make use of the laptop or web-cam and by recognizing the hand
gesture we could control mouse and perform basic operations like mouse pointer
controlling, select and deselect using left click, and a quick access feature for file
transfer between the systems connected via network LAN cable. The project done
is a “Zero Cost” hand recognition system for laptops, which uses simple
algorithms to determine the hand, hand movements and by assigning an action
for each movement.

But we have mainly concentrated on the mouse pointing and clicking actions
along with an action for the file transfer between connected systems by hand
action and the movements. The system we are implementing which is been
written in python code be much more responsive and is easily implemented since
python is a simple language and is platform independent with a flexibility and is
portable which is desirable in creating a program which is focused in such an aim
for creating a Virtual Mouse and Hand Recognition system. The system be much
more extendable by defining actions for the hand movement for doing a specific
action. It could be further modified to any further extent by implementing such
actions for the set of hand gestures, the scope is restricted by your imagination .
Block Diagram / System Architecture
1.Capturing real time video using Web-Camera: We will need a sensor to detect
the user's hand movements in order for the system to work. As a sensor, the
computer's webcam is used. The webcam records real-time video at a fixed frame
rate and resolution determined by the camera's hardware. If necessary, the
system's frame rate and resolution can be changed.
2.Converting the video captured into HSV format: The video has also been
converted into HSV (hue, saturation, meaning, also called HSB), an alternative
representation of the RGB color model created by computer graphics researchers
to better reflect the perception of colored characteristics by human vision.
3) Each image frame is processed separately: Following the capture of the video,
it goes through a brief pre-processing stage before being processed one frame at
a time.
4) Conversion of each frame to a greyscale image: A grayscale image has a lower
computational complexity than a colored image. It also aids in faster color
calibration without the use of external noise. All the necessary operations were
carried out after the image was converted to grayscale.
5) Calibrate the color ranges: The device enters the calibration mode after the
above steps, which assigns each color according to the HSV rule to its color hue,
saturation or value values. Every color already has its predetermined values. For
accurate color detection, the user can adjust the ranges. In the diagram below
you can clearly see the variety of values used to detect every color cap.
6) Calculate the image's centroid by locating the image's region. To guide the
mouse pointer, the user must first choose a point whose coordinates can be sent
to the cursor. The device can monitor cursor movement using these coordinates.
As the object travels around the frame, these coordinates change over time.
7) Tracking the mouse pointer. After determining the coordinates, the mouse
driver is accessed and the coordinates are sent to the cursor. The cursor positions
itself in the required position using these coordinates. As a result, the mouse
moves proportionally across the screen as the user moves his hands across the
camera's field of view.
8) Simulating the mouse actions. To create the control actions in simulation
mode, the user must make hand gestures. The computation time is reduced due
to the use of colour pointers.
Work Schedule
 
Month. SEM-I SEM-II

1.  Project Title Selection,  

Objectives

2. Synopsis    

3.    

4.   Participate in Project Competition,

PaperPublication 
Required Hardware and Software
A)Hardware Requirement: -
The following describes the hardware needed in order to execute and develop
the Virtual Mouse application: • Computer Desktop or Laptop The computer
desktop or a laptop will be utilized to run the visual software in order to display
what webcam had captured. A notebook which is a small, lightweight and
inexpensive laptop computer is proposed to increase mobility of the
application. System will be using Processor: Core2Duo Main Memory: 2 GB
RAM (Minimum) Hard Disk: 512 GB (Minimum) Display: 14" Monitor (For more
comfort) Webcam Web cam is utilized for image processing, the webcam will
continuously taking image in order for the program to process the image and
find pixel position.
B)Software Requirement: -
The following describes the software needed in-order to develop the Virtual
Mouse application: Python: To access camera & tracking all hand motion,
python is very easy & accurate to use. Python comes with lots of build in
libraries which makes code short and easily understandable. Python version
required for building of this application is 3.7 Open CV Library: OpenCV are also
included in the making of this program. OpenCV (Open-Source Computer
Vision) is a library of programming functions for real time computer vision.
OpenCV have the utility that can read image pixels value, it also has the ability
to create real time eye tracking and blink detection. Tkinter: The Tkinter
package is the standard Python interface to the Tk GUI toolkit. Both Tk and
Tkinter are available on most Unix platforms, as well as on Windows systems.
To make UI for application we used Tkinter.
Advantages & Disadvantages
Advantages:-
 The system is easy to install.
 It can be used as easy mouse control for users.
 It is not an electronic based system so one can easily make use of laptops to
install this system.
 It is cost-effective.
 It maximizes accuracy and reduces energy usage.

Disadvantages:-
 It requires a lot of memory.
 The alarm it has limited accuracy.
Application
 After this COVID-19 situation, it is not safe to use the devices by
touching them because it may result in a possible situation of spread of
the virus by touching the devices, so the proposed AI virtual mouse can
be used to overcome these problems since hand gesture and hand Tip
detection is used to control the PC mouse functions by using a webcam
or a built-in camera.
 AI Virtual Mouse. Developed an AI-based approach for controlling the
mouse movement using Python and openCV with real-time camera that
detects hand landmarks, tracks gesture patterns instead of a physical
mouse.
 The proposed AI virtual mouse system can be used to overcome
problems in the real world such as situations where there is no space to
use a physical mouse and also for the persons who have problems in their
hands and are not able to control a physical mouse.
References
[1] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj,” Mouse Control
using a Web Camera based on Color Detection” ,IJCTT, vol.9, Mar 2014
[2] Angel, Neethu.P.S,”Real Time Static & Dynamic Hand Gesture Recognition”,
International Journal of Scientific & Engineering Research Volume 4, Issue3,
March-2013.
[3] Chen-Chiung Hsieh and Dung-Hua Liou,” A Real Time Hand Gesture
Recognition System Using Motion History Image”icsps, 2010 [4] Hojoon Park, “A
Method for Controlling the Mouse Movement using a Real Time Camera”, Brown
University, Providence, RI, USA, Department of computer science, 2008.