ARTIFICIAL INTELLIGENCE-BASED GESTURE

CONTROL ROBOT
By
ADIL GULZAR 2019-GCUF- 057567
EMMAD-UD-DIN 2019-GCUF-057561
MUHAMMAD HAMMAD IQBAL 2019-GCUF-057553

BACHELOR OF SCIENCE
IN
COMPUTER SCIENCE

DEPARTMENT OF COMPUTER SCIENCE

Government College University Faisalabad
2023
Government College University Faisalabad 2
Table of Contents

Table of Contents...................................................................................................................2
Revision History…..................................................................................................................3
1. Introduction…....................................................................................................................4
1.1 General Information….............................................................................................4
1.2 Purpose......................................................................................................................4
1.3 Document Conventions….........................................................................................4
1.4 Project Objective…...................................................................................................4
1.5 Intended Audience.....................................................................................................5
1.6 Project Scope............................................................................................................5
2. Overall Description............................................................................................................6
2.1 Product Perspective...................................................................................................6
2.2 Product Features.......................................................................................................6
2.3 User Classes and Characteristics…...........................................................................6
2.4 Design and Implementation Constraints...................................................................6
2.5 Operating Environment.............................................................................................6
2.6 User Documentation….............................................................................................6
2.7 Assumption and Dependencies.................................................................................7
3. System Features..................................................................................................................7
3.1 System Features........................................................................................................8
4. External Interfaces Requirements....................................................................................8
4.1 User Interface...........................................................................................................8
4.2 Hardware Interface....................................................................................................8
4.3 Software Interface.....................................................................................................8
4.4 Communication Interface.........................................................................................8
5. Other Nonfunctional Requirements….............................................................................8
5.1 Performance Requirements.......................................................................................8
5.2 Safety Requirements….............................................................................................8
5.3 Security Requirements..............................................................................................8
5.4 Software Quality Attributes......................................................................................8
6. WBS Project Management................................................................................................9
7. Tools & Technologies…....................................................................................................17
7.1 Programming Language….......................................................................................17
7.2 Database/Data storage….........................................................................................17
7.3 Operatingsystem.......................................................................................................18
Appendix A: Glossary….......................................................................................................18
Appendix B: Check List........................................................................................................19
Appendix C: Supervisory Committee..................................................................................20
 Government College University Faisalabad 3

Revision History

Name Date Reason for Changes Version

 Government College University Faisalabad 4

1. Introduction

1.1 General Information

Project Name: Artificial intelligence-based hand control robot. Starting Date:
Controlling Agency: Final Date:
Prepared By: Adil Gulzar, Emmad-Ud-Din, M. Hammad Iqbal Authorized by:

1.2 Purpose
An artificial intelligence-based hand control robot has various purposes and applications.
Some of the key purposes include:
1. Assistive Robotics: These robots are designed to help individuals with limited hand mobility
or disabilities. AI algorithms enable the robot to interpret and respond to hand gestures or
commands, allowing users to interact with the robot and perform tasks that would otherwise be
challenging.
2. Rehabilitation and Therapy: AI-controlled hand robots can be used in rehabilitation
programs for patients recovering from injuries or surgeries. The robot can provide targeted
exercises and activities to improve hand strength, dexterity, and coordination under the
guidance of a therapist.

1.3 Document Conventions

When documenting an artificial intelligence-based hand control robot, it is important to
follow certain conventions to ensure clarity and consistency. Here are some document
conventions that can be helpful:

1. Title and Header: Begin the document with a clear and descriptive title that reflects the
purpose or functionality of the robot. Use headers and sub headers to organize the content and
make it easily scannable.
2. Introduction: Provide an overview of the document, introducing the robot, its purpose, and
the key features it offers. Explain the problem it aims to solve or the tasks it is designed to
perform.
 Government College University Faisalabad 5
3. Architecture and Components: Describe the overall architecture of the robot, including the
hardware components (e.g., robotic arm, sensors, actuators) and the software components (e.g.,
AI algorithms, control systems, user interfaces). Provide diagrams and illustrations to enhance
understanding.

By following these document conventions, you can create a comprehensive and well-structured
document that effectively communicates the key aspects of an AI-based hand control robot.

1.4 Project Objective

The objective of an artificial intelligence-based hand control robot project can
vary depending on the specific goals and intended application. However, here are some
common objectives that such a project may aim to achieve:

1. Develop a Robust Hand Control System: The primary objective is to create a reliable and
efficient hand control system that utilizes artificial intelligence algorithms to interpret and
respond to hand gestures or commands accurately. The goal is to enable seamless and intuitive
control of the robot's hand movements.

2. Enhance Accessibility and Assistive Capabilities: One of the key objectives is to improve
accessibility for individuals with limited hand mobility or disabilities. The project aims to
develop a robotic system that assists users in performing daily tasks, such as grasping objects,
manipulating tools, or interacting with their environment more independently.

3. Enable Prosthetic Hand Functionality: If the project focuses on prosthetics, the objective is
to create a highly functional artificial hand that closely mimics the capabilities of a natural
hand. The AI-based hand control system aims to provide precise and intuitive control, allowing
users to perform a wide range of tasks with their prosthetic hand.

It's important to note that the specific objectives of an AI-based hand control robot project will
depend on the intended use case and the needs of the target audience.
 Government College University Faisalabad 6

1.5 Intended Audience and Reading Suggestions

The document on an artificial intelligence (AI)-based hand control robot is intended for a
diverse audience interested in robotics, artificial intelligence, human-robot interaction, and
related fields. The primary target audience includes:

1. Researchers and Engineers: Professionals and researchers in the fields of robotics, artificial
intelligence, and human-robot interaction will find the document valuable. It provides insights
into the development, implementation, and advancements of AI-based hand control robots,
offering a resource for their research and development endeavors.

2. Developers and Technologists: Individuals involved in the development of AI-based hand

control systems, whether in academia or industry, can benefit from the document. It provides
technical information on hardware components, AI algorithms, control systems, and integration
techniques, assisting them in designing and building advanced hand control robots.

3. Healthcare Professionals and Therapists: The document caters to healthcare

professionals, therapists, and rehabilitation specialists working with individuals with hand
disabilities or limitations. It highlights the potential benefits and applications of AI-based hand
control robots in rehabilitation and therapy programs, providing insights into how these
technologies can improve patient outcomes.
For readers interested in further exploration of the topic, here are some suggested readings and
resources:

1.6 Project Scope

The scope of an artificial intelligence (AI)-based hand control robot project typically
encompasses various aspects involved in designing, developing, and deploying the robot. Here
are some key components that generally fall within the project scope:
 Government College University Faisalabad 7
1. System Design and Architecture: Define the overall system design and architecture of
the AI-based hand control robot. This includes determining the hardware components, such as
robotic arm, sensors, actuators, and the software components, such as AI algorithms, control
systems, and communication interfaces.

2. Hand Gesture Recognition: Develop AI algorithms or machine learning models to analyze

and interpret hand gestures or commands accurately. This involves training the algorithms using
appropriate datasets, optimizing their performance, and integrating them into the control system.

3. Robotic Hand Control: Implement control mechanisms that translate the interpreted
hand gestures or commands into corresponding movements of the robotic hand. This
involves developing algorithms for grasping, releasing, and manipulating objects based on
the user's input.

It is essential to note that the specific scope of an AI-based hand control robot project may vary
depending on the project's objectives, resources, and constraints. Therefore, careful planning and
scoping are necessary to define the project's boundaries and ensure successful execution.

2. Overall Description

2.1 Product Perspective

1. AI-Powered Hand Control: The robot utilizes advanced AI algorithms to interpret
and analyze hand gestures and commands. This allows for seamless and natural interaction
between the user and the robot.

2. Dexterous Robotic Hands: The robot is equipped with highly dexterous robotic
hands that can replicate a wide range of human hand movements. These hands may feature
multiple degrees of freedom, allowing for intricate grasping, gripping, and manipulation of
objects.
 Government College University Faisalabad 8
3. Gesture Recognition: The AI system is trained to recognize and interpret various
hand gestures and motions made by the user. This enables the user to control the robot
intuitively, without the need for complex programming or explicit instructions.

Overall, an artificial intelligence-based hand control robot offers an innovative solution for
intuitive and precise control of robotic hands. It combines advanced AI algorithms, dexterous
hands, and intuitive interaction to enable a wide range of applications in various industries, while
ensuring safety and adaptability.

2.2 Product Features

Sure! Here are some specific product features of an artificial intelligence-based hand control
robot:

1. Hand Gesture Recognition: The robot uses AI algorithms to recognize and interpret hand
gestures made by the user. It can detect and differentiate between various gestures, such as
grasping, pinching, pointing, and releasing.

2. Intuitive Control Interface: The robot provides an intuitive control interface that allows
users to control the robotic hand using natural hand movements. This eliminates the need for
complex manual programming or external controllers.

3. Precise Manipulation: The robotic hand is designed to perform precise manipulation tasks
with high accuracy. It can grasp and manipulate objects of different shapes, sizes, and
materials, allowing for a wide range of applications.

These features collectively make an artificial intelligence-based hand control robot a versatile
and efficient tool for precise manipulation tasks, offering intuitive control, adaptive learning, and
integration possibilities for various applications.
 Government College University Faisalabad 9
2.3 User Classes and Characteristics
User Classes and Characteristics of an artificial intelligence-based hand control robot can
vary depending on the specific application and target audience. Here are some potential user
classes and their corresponding characteristics:

1. Researchers and Developers:

- Technical background in robotics, AI, or related fields.
- Familiarity with programming languages and algorithms.
- Interest in exploring and advancing the capabilities of robotic hand control.
- Willingness to experiment, develop new techniques, and contribute to the field.

2. Industrial Professionals:
- Engineers and technicians working in manufacturing or automation industries.
- Need for precise and dexterous manipulation of objects in industrial processes.
- Desire to enhance productivity, accuracy, and efficiency in manufacturing tasks.
- Ability to integrate the robot into existing production systems.

3. Healthcare Practitioners:
- Surgeons, doctors, and medical professionals.
- Use of robotic hands for surgical procedures, prosthetics, or rehabilitation.
- Need for precise control and manipulation in delicate medical tasks.
- Interest in leveraging AI and robotics to improve patient outcomes and healthcare practices.

4. Educational Institutions:
- Teachers, students, and researchers in robotics or AI education.
- Incorporation of robotic hand control in curriculum or research projects.
- Interest in hands-on learning and experimentation.
- Need for user-friendly interfaces and educational resources.

These user classes and characteristics represent a broad range of potential users who can benefit
from an artificial intelligence-based hand control robot. Understanding the specific needs, skills,
 Government College University Faisalabad 10
and goals of these user classes can help in designing and developing a product that caters to their
requirements effectively.

2.4 Operating Environment

The operating environment of an artificial intelligence-based hand control robot can vary
depending on its intended application. Here are some common operating environments
where such a robot may be used:

1. Industrial Settings:
- Manufacturing facilities: The robot can be deployed in industries such as automotive,
electronics, or assembly lines, where precise manipulation and control of objects are required.
- Warehouses and logistics: The robot can assist in picking and packing tasks, inventory
management, or material handling.

2. Healthcare and Medical Environments:

- Operating rooms: The robot may be used in surgical procedures, assisting surgeons
with delicate tasks or providing precision in minimally invasive surgeries.
- Rehabilitation centers: The robot can aid in physical therapy or assist individuals with
mobility impairments in performing daily activities.

3. Research and Development Laboratories:

- Robotics research labs: The robot can be utilized by researchers and developers for
experimentation, testing new algorithms, or advancing the capabilities of robotic hand
control.
- AI and machine learning labs: The robot can serve as a platform for developing and
training AI models related to gesture recognition, object manipulation, or adaptive learning.

It's important to note that the operating environment may have specific requirements and
considerations, such as safety protocols, cleanliness standards, or connectivity infrastructure,
depending on the application and industry. Adhering to these requirements ensures the optimal
 Government College University Faisalabad 11
performance and integration of the artificial intelligence-based hand control robot in its intended
operating environment.

2.5 Design and Implementation Constraints

Design and implementation constraints play a crucial role in the development of an
artificial intelligence-based hand control robot. Here are some common constraints that need to
be considered during the design and implementation process:

1. Physical Constraints:
- Size and Weight: The robot should be designed to have a suitable form factor and weight to
ensure it can operate effectively and be integrated into various environments.
- Ergonomics: The design should consider human ergonomics, ensuring that the robot can be
comfortably operated and controlled by users.

2. Safety Constraints:
- Collision Avoidance: The robot should incorporate sensors and algorithms to detect and
avoid collisions with objects or humans in its operating environment.
- Force Limiting: The robot should have mechanisms in place to limit the applied force
and prevent injuries or damage during interactions with objects or individuals.
- Emergency Stop: The robot should include an emergency stop mechanism that can
be activated in case of any unforeseen situations or hazards.

3. Power and Energy Constraints:

- Power Efficiency: The robot should be designed to optimize power consumption to prolong
battery life or reduce energy requirements when connected to a power source.
- Charging or Battery Replacement: Considerations should be made for charging or
battery replacement mechanisms to ensure uninterrupted operation.

These constraints must be carefully evaluated and addressed during the design and
implementation phases to create an artificial intelligence-based hand control robot that is safe,
efficient, reliable, and meets the specific requirements of its intended application and users.
 Government College University Faisalabad 12

2.6 User Documentation

User documentation for an artificial intelligence-based hand control robot is essential to
provide users with the necessary information to operate and utilize the robot effectively. Here are
some key components that can be included in the user documentation:

1. User Manual:
- Overview: Provide an introduction to the robot, its capabilities, and the purpose of the user
manual.
- Safety Instructions: Clearly outline safety precautions, including guidelines for operating
the robot safely and preventing potential hazards.
- Getting Started: Step-by-step instructions on how to set up the robot, including assembly,
connectivity, and power requirements.
- User Interface: Explain how to interact with the control interface, including
gestures, commands, or other input methods.
- Robot Operation: Detail the procedures and techniques for controlling the robotic
hand, including hand movements, gestures, or button commands.
- Maintenance and Care: Provide guidelines for cleaning, maintenance, and proper storage
of the robot to ensure its longevity and optimal performance.
- Troubleshooting: Include a troubleshooting section with common issues and their solutions.
- Technical Specifications: List detailed specifications of the robot, including dimensions,
weight, power requirements, and any compatibility information.

2. Quick Start Guide:

- Offer a condensed version of the user manual, providing a quick and easy-to-follow guide for
users who want to get started without delving into the full manual.
- Highlight the essential steps to set up and operate the robot, focusing on the most common
tasks or features.

3. FAQs and Troubleshooting:

 Government College University Faisalabad 13
- Compile a list of frequently asked questions (FAQs) to address common queries and
concerns users may have.
- Provide troubleshooting tips and solutions for known issues or errors that users may
encounter during operation.
- Include contact information for customer support or technical assistance if additional help
is needed.

The user documentation should be written in clear and concise language, with a user-centric
approach. It should be easily accessible and available in both digital and printed formats,
ensuring that users can refer to it whenever necessary. Regular updates to the documentation
may be required to reflect changes in the robot's features, improvements, or bug fixes.

2.7 Assumptions and Dependencies

When designing and implementing an artificial intelligence-based hand control robot,
several assumptions and dependencies are made to ensure its functionality and performance.
Here are some common assumptions and dependencies:

1. Gesture Recognition Accuracy: The robot assumes that the AI algorithms and sensors used
for gesture recognition will accurately interpret and classify hand gestures performed by the
user. The system relies on a certain level of accuracy to correctly understand user commands and
translate them into actions.

2. Adequate Training Data: The AI algorithms used for gesture recognition and
manipulation require a sufficient amount of diverse and representative training data. It is
assumed that the training data used to develop and refine the algorithms adequately captures
the variability and nuances of hand movements.

3. Proper Lighting Conditions: The robot assumes that the operating environment will have
adequate lighting conditions for optimal performance. Poor lighting conditions, such as low light
or harsh glare, may affect the accuracy of gesture recognition or object manipulation.
 Government College University Faisalabad 14
These assumptions and dependencies are taken into consideration during the design,
development, and use of an artificial intelligence-based hand control robot. Understanding and
addressing these factors are crucial to ensuring the reliability, accuracy, and usability of the robot
in real-world applications.

3. System Features
An artificial intelligence-based hand control robot can have a range of system features to
enable effective control and manipulation of objects. Here are some common system features:

3.1) Gesture Recognition:

- The ability to recognize and interpret hand gestures performed by the user.
- Utilization of computer vision techniques, machine learning algorithms, or depth sensors
to accurately recognize and classify gestures.

3.2) Object Detection and Tracking:

- Capabilities to detect and track objects in the robot's environment.
- Integration of sensors, such as cameras or depth sensors, to perceive and locate objects
for manipulation.

3.3) Dexterous Hand Manipulation:

- Precise control over the robot's hand movements and manipulation capabilities.
- Utilization of multi-fingered grippers or robotic hands capable of various degrees of
freedom.
- Fine-grained control for tasks such as grasping, picking, lifting, rotating, and
releasing objects.

3.4) Adaptive Learning:

- Ability to learn and adapt to user preferences and hand movement patterns.
- Incorporation of machine learning algorithms that enable the system to improve its
performance over time through user interactions.
 Government College University Faisalabad 15
3.5) Natural Language Processing (NLP) Integration:
- Integration of NLP capabilities to enable verbal or textual commands for controlling
the robot.
- Speech recognition or natural language understanding algorithms for interpreting
and executing user instructions.

These system features enable the artificial intelligence-based hand control robot to accurately
interpret user commands, manipulate objects with dexterity, and provide a seamless and
immersive user experience. The specific features and their implementation may vary depending
on the application, target audience, and technological capabilities of the robot.

4. External Interfaces Requirements

When designing an artificial intelligence-based hand control robot with external
interfaces, there are several requirements to consider. These requirements will depend on the
specific use case and functionality of the robot. Here are some general requirements to consider:

4.1.) Sensor Interfaces: The robot should have interfaces to connect and communicate with
various sensors. These may include cameras, depth sensors, force sensors, or any other sensors
used for perception and feedback.

4.2.) Actuator Interfaces: The robot's hand control system may require interfaces to connect
and control different types of actuators, such as motors or servos, for hand movements and
grasping. These interfaces should provide the necessary power and control signals to actuate the
hand effectively.

4.3.) Communication Interfaces: The robot needs interfaces for communication with external
devices or systems. This could include wired or wireless interfaces such as Ethernet, USB,
Bluetooth, or Wi-Fi. These interfaces enable data exchange, control commands, and status
updates between the robot and external systems.
 Government College University Faisalabad 16
4.4.) API and Protocol Support: The robot should support standard APIs (Application
Programming Interfaces) and protocols to facilitate integration with other software or hardware
systems. Commonly used APIs for robotics include ROS (Robot Operating System) or specific
vendor-specific APIs.

It's worth noting that these requirements serve as a general starting point and should be tailored
to the specific application and environment where the hand control robot will be used.
5. Other Nonfunctional Requirements

5.1.) Performance: The system should have real-time responsiveness and provide smooth and
precise control over the robot hand. It should minimize latency and ensure accurate execution of
commands.

5.2.) Reliability: The system should be reliable, with minimal downtime or failure rates. It
should be able to recover from errors gracefully and resume normal operation without causing
harm or damage.

5.3.) Safety: Safety is paramount when designing a robot that interacts with humans. The
system should adhere to safety standards and regulations, ensuring that the robot hand operates
in a safe manner, avoiding potential hazards or accidents.

5.4.) Security: As with any system involving artificial intelligence, data security and privacy are
crucial. The system should implement appropriate security measures to protect user data,
prevent unauthorized access, and ensure the integrity and confidentiality of information.

These nonfunctional requirements aim to ensure the overall performance, safety, reliability,
and usability of the artificial intelligence-based hand control robot, while also addressing
important ethical and security considerations.
 Government College University Faisalabad 17
6) WBS Project Management

The Robot Company

Mission, System and Test Electronics and control Design and Manufacturing

System design and integration

MCU subsystem and control 3D Models and simulation

Manufacture parts and Assemblies

Mission and Application Sensors, Actuators and Power

Support Equipment BTE, GSE, jigs

Interface design and source tracking PCB design, Layout and Assembly
 Government College University Faisalabad 18

Diagrams
Class Diagram:

Start Accelerometer Arduino Nano

Convert data into

angles 0-450°

Rf Receiver Rf Transmitter

Arduino Uno

Checking
of angles

Forward Back Left Right

End Car movement Motor shield

 Government College University Faisalabad 19

Activity Diagram:

Start

Input from Input from Input from

contact sensor flex sensor accelerometer

Gesture sensing algorithm

Control integer selection

If gesture If
If backward gesture right
If
gesture forward gesture left

Send x = 1 Send x = 4
Send x = 1 Send x = 3
 Government College University Faisalabad 20

Sequence Diagram:

Robot arm Ref robot collaboration

Robot Cutting tool
Us
Controll
er
Setup System er

Composed life line

Referenced interaction fragment

(Explicit) return
 Government College University Faisalabad 21

Component diagram:

Hand Accelerometer

Arduino Nano

RF Transmitter RF Receiver

Arduino Uno

Motor shield

Car
 Government College University Faisalabad 22

Logic Sequence Diagram:

Receiver:

6V Battery
RF receiver

Diode Decoder

Driver IC
Series MC

Motor Motor
 Government College University Faisalabad 23

User Case Diagram:

Calibrate System

Present to audience
Uses
M
Uses
Uses Uses Extends
Update
Remain still
keyframe

Extends
Perform Extract static
gesture and dynamic

User
Recognize
gesture
 Government College University Faisalabad 24

ER Diagram:
Start

GPS Position estimator

installation

Read path info Tracking trajectory

Create path by manual Path

driving Last point in trajectory?
created

The path is Position Initialization

determined
?

Chose next tracking point

Arrived
initial position

Same path

Chose first tracking point

End
 Government College University Faisalabad 25

DFD Diagram:
Remote control Robot

Remote control Wheels

Motors Movement

RF transmitter
Microcontroller

Direction

T-X Antenna
RF receiver

R-X Antenna
 Government College University Faisalabad 26

Appendix A: Glossary
• Purpose: AI based gesture control robots are designed to help individuals
with limited hand mobility or disabilities. AI algorithms enable the robot to
interpret and respond to hand gestures or commands, allowing users to
interact with the robot and perform tasks that would otherwise be
challenging.

• Project Description: The robot utilizes advanced AI algorithms to interpret

and analyze hand gestures and commands. This allows for seamless and
natural interaction between the user and the robot.

• Project Scope: The scope of an artificial intelligence (AI)-based hand control

robot project typically encompasses various aspects involved in designing,
developing, and deploying the robot. Here are some key components that
generally fall within the project scope.

• User Characteristics: Characteristics of an artificial intelligence-based hand

control robot can vary depending on the specific application and target
audience.

• System features: The ability to recognize and interpret hand gestures

performed by the user, integration of different sensors.

• Tools: Accelerometer, Arduino UNO board, 12volt battery, RF transmitter

module, RF receiver, Decoder, Motor driver, Microcontroller, Wires, Tires,
Switch, Hand gloves.
 Government College University Faisalabad 27

Appendix B: Check List

Check list Yes No

Starting/Ending dates

Purpose

Project description

Project scope

User characteristics

System features

Tools
 Government College University Faisalabad 28

Appendix C: Supervisory Committee

For approval of any two consultant Teachers

Name: Name:

Designation: Designation:

Comments: Comments:

Signature: Signature:

(For Office Use Only)

Approved Date:
Meeting Required Date: Group ID:
Rejected Time:
Place: _

Remarks

Project Title (If revised):

Project Coordinator