Workpiece Classification Using Robot Arm and Computer Vision

THE UNIVERSITY OF DA NANG
UNIVERSITY OF SCIENCE AND TECHNOLOGY

FACULTY OF ADVANCED SCIENCE AND TECHNOLOGY
GRADUATION THESIS
MAJOR: INDUSTRIAL INFORMATICS
THESIS TITLE:
WORKPIECE CLASSIFICATION SYSTEM

USING ROBOT ARM AND
COMPUTER VISION
STUDENTS: TRUONG THI MAI HUE, 122170072

NGUYEN VAN ANH QUAN, 122170042
PHAN CHI VU, 122170068
CLASS: 17PFIEV2
Supervisor: Dr. NGUYEN LE HOA

Co-supervisor: Mr. LE VAN DAY
Da Nang, 07/2022
REVIEWER’S COMMENT
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
SUPERVISOR’S COMMENT
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
...................................................................................................................................................
ABSTRACT
The thesis was carried out in Oanh Vu Electronics Co., Ltd. The project aims to
build a robotics system combined with computer vision to classify workpieces on the
conveyor, effectively improving productivity. The thesis focuses on the study of robot
arms that pick up moving workpieces on the conveyor and classify them according to
selected criteria.
The contents of the thesis include the research and design of a simple robot arm
that picks up workpieces moving on a conveyor based on pre-existing requirements,
using Arduino to control and classify workpieces based on color through the image
processing program on the laptop.
i
DANANG UNIVERSITY
UNIVERSITY OF SCIENCE AND SOCIALIST REPUBLIC OF VIET NAM
TECHNOLOGY Independence – Freedom - Happiness
FACULTY OF ADVANCED
SCIENCE AND TECHNOLOGY
GRADUATION PROJECT REQUIREMENTS
Index Student Student No. Class Major

1 Truong Thi Mai Hue 122160072 17PFIEV2 Industrial informatics
2 Nguyen Van Anh Quan 122170042 17PFIEV2 Industrial informatics
3 Phan Chi Vu 122170068 17PFIEV2 Industrial informatics
1. Name of the project:

“Workpiece classification system using robot arm and computer vision”.
2. Project topic: ☐ has signed intellectual property agreement for the final result
3. Initial figure and data:
Initial figures and data are generated by students and information is searched on
the Internet and references.
4. Project contents:
Chapter 1: PROJECT OVERVIEW - This chapter introduces an overview of the
project, its purpose, and its objective.
Chapter 2: DESIGN A CLASSIFICATION SYSTEM USING ROBOT ARM
AND COMPUTER VISION - This chapter describes the design and selection of each
part of the system.
Chapter 3: CONTROL ALGORITHM - This part presents the processing
algorithms used in the project.
Chapter 4: EXPERIMENTAL RESULTS AND EVALUATION - This chapter
gives the experimental results and evaluation of the system's operation.
Chapter 5: CONCLUSION - This chapter presents the results achieved, the
remaining limitations of the system and orientations for development.
5. Drawings, charts:
- Workpiece drawing
- Workpiece container drawing
- Control box drawing
- Robot arm drawing
ii
- Robot arm detailed drawing
- System model drawing
6. Supervisors:
NGUYEN Le Hoa, Dr.
Faculty of Advanced Science and Technology, University of Science and
Technology - The University of Danang, Vietnam.
LE Van Day, Engineer
CEO, Oanh Vu Electronics Co., Ltd.
7. Date of assignment: 7/3/2022
8. Date of completion: 5/7/2022
Da Nang, July 5th, 2022
Head of Division Supervisor
iii
ACKNOWLEDGEMENT
With the permission of the leadership of Oanh Vu Electronics Co., Ltd as well as
the Faculty of Advanced Science and Technology, The Danang University - University
of Science and Technology, we had the chance to practice doing a graduation project at
Oanh Vu Electronics Co., Ltd and this is a valuable opportunity for us to work in a
professional environment of enterprises and apply the theoretical knowledge of the
major into practice. Below is the graduation internship report, the result of studying at
Oanh Vu Electronics Co., Ltd.
We would like to say many thanks to all employees and managers in Oanh Vu
Electronics Co., Ltd and our supervisor, Dr. Nguyen Le Hoa for the help and enthusiastic
instruction during our internship.
Thank you sincerely!
iv
COMMITMENT
We are currently practicing and working on a graduation project at Oanh Vu

Electronics Co., Ltd. We pledge to perform well following the academic integrity
regulations of The Danang University - University of Science and Technology (UD -
DUT) and Decision No. 29 / QD-DHBK dated January 9, 2017 of the Rector of UD -
DUT. If violation, we will be responsible for the full responsibility and receive the level
of violation handling following the university's regulations.
Students
Truong Thi Mai Hue Nguyen Van Anh Quan Phan Chi Vu
v
TABLE OF CONTENTS
ABSTRACT .....................................................................................................................i
GRADUATION PROJECT REQUIREMENTS............................................................ ii
ACKNOWLEDGEMENT ..............................................................................................iv
COMMITMENT .............................................................................................................v
TABLE OF CONTENTS ...............................................................................................vi
LIST OF TABLES ...................................................................................................... viii
LIST OF FIGURES ........................................................................................................ix
LIST OF ABBREVIATIONS ......................................................................................xi
Chapter 1. PROJECT OVERVIEW ............................................................................1
1.1. Introduction ..........................................................................................................1
1.2. Related research....................................................................................................2
1.3. Research objective ................................................................................................3
Chapter 2. DESIGN A CLASSIFICATION SYSTEM USING ROBOT ARM AND
COMPUTER VISION .....................................................................................................4
2.1. System overview ..................................................................................................4
2.2. Workpiece selection and design ...........................................................................4
2.3. Conveyor selection ...............................................................................................5
2.4. Robot arm design ..................................................................................................6
2.4.1. Robot arm structure selection ................................................................ 6
2.4.2. Kinematics problem ...............................................................................8
a) Denavit - Hertenberg representation ................................................................ 8
b) Forward kinematics ........................................................................................10
c) Inverse kinematics .......................................................................................... 11
2.4.3. Transmission system selection for rotational joints ............................ 13
2.4.4. End-effector selection ..........................................................................13
2.4.5. Material and dimension selection ........................................................16
2.5. Control circuit design .........................................................................................18
2.5.1. Central processing block .....................................................................18
2.5.2. Communication block..........................................................................19
a) UART communication ...................................................................................19
b) I2C communication ........................................................................................19
2.5.3. Power supply block .............................................................................20
vi
a) Power supply for Arduino ..............................................................................20
b) Power supply for geared motor and solenoid valve .......................................21
c) Power supply for servo and vacuum pump ....................................................22
2.5.4. Capacity block .....................................................................................23
a) Servo control capacity block ..........................................................................23
b) Geared motor control capacity block ............................................................. 23
c) Vacuum pump and solenoid valve control capacity block............................. 24
2.5.5. Complete control circuit schematic .....................................................25
2.6. Camera selection ................................................................................................28
2.7. Complete model of the system ...........................................................................28
Chapter 3. CONTROL ALGORITHM .....................................................................29
3.1. The main idea .....................................................................................................29
3.2. Image processing algorithm ...............................................................................29
3.3. Robot arm control algorithm ..............................................................................33
Chapter 4. EXPERIMENTAL RESULTS AND EVALUATION ........................... 35
4.1. Experimental results ...........................................................................................35
4.1.1. Control circuit ......................................................................................35
4.1.2. Robot arm ............................................................................................ 36
a) Joint movement .............................................................................................. 36
b) Impact of vacuum gripper ..............................................................................36
c) Robot arm control ........................................................................................... 37
4.1.3. Workpiece detection algorithm ........................................................... 38
4.1.4. Classification system model ................................................................ 39
4.2. Experimental evaluation .....................................................................................41
4.2.1. Operation of control circuit .................................................................41
4.2.2. Operation of robot arm ........................................................................41
a) Movement range and end-effector impact .....................................................41
b) Robot arm control .......................................................................................... 41
4.2.3. Workpiece detection algorithm ........................................................... 41
4.2.4. Operation of classification system model ............................................41
CONCLUSION .............................................................................................................42
REFERENCES ..............................................................................................................44
APPENDIX .....................................................................................................................1
vii
LIST OF TABLES
Table 2.1. Technical specifications of geared motor 370L 24VDC................................ 6

Table 2.2. Denavit - Hertenberg parameters .................................................................10
Table 2.3. Technical specifications of MG996R servo .................................................13
Table 2.4. Technical specifications of SG90 servo .......................................................14
Table 2.5. Technical specifications of ZPT vacuum pad ..............................................15
Table 2.6. Technical specifications of vacuum pump PYP-370 ...................................15
Table 2.7. Technical specifications of solenoid valve YQK513 ...................................15
Table 2.8. Technical specifications of Arduino Nano ...................................................19
Table 2.9. Technical specifications of LCD 1602 .........................................................20
Table 2.10. Technical specifications of I2C communication converter module ...........20
Table 2.11. Technical specifications of Adapter 5VDC - 1A .......................................21
Table 2.12. Technical specifications of Adapter 24VDC - 2A .....................................22
Table 2.13. Technical specifications of LM2596 module .............................................22
Table 2.14. Technical specifications of webcam Logitech C310..................................28
Table 4.1. Joints movable range of robot arm ............................................................... 36
Table 4.2. Experimental results of robot arm control....................................................37
Table 4.3. Experimental results of workpiece detection algorithm in different lighting
conditions.......................................................................................................................39
Table 4.4. Experimental results of workpiece classification system in different lighting
conditions.......................................................................................................................40
viii
LIST OF FIGURES
Figure 1.1. Project " Thiết kế và chế tạo cánh tay robot phân loại sản phẩm sử dụng xử
lý ảnh " by UD - DUT students .......................................................................................2
Figure 1.2. Project "Thiết kế và chế tạo cánh tay robot 3 bậc tự do ứng dụng trong phân
loại sản phẩm bằng xử lý ảnh" by UD - DUT students ...................................................2
Figure 1.3. Functional block diagram of the system .......................................................3
Figure 2.1. Shape and size of the workpiece design .......................................................4
Figure 2.2. Shape and size of the workpiece container ...................................................5
Figure 2.3. Conveyor with geared motor 370L 24VDC..................................................6
Figure 2.4. Some popular robot arm structures ............................................................... 7
Figure 2.5. The overall structure of the robot arm ..........................................................7
Figure 2.6. Modeling links and joints in terms of Denavit - Hartenberg ........................8
Figure 2.7. Identify the coordinate system for the robot arm ..........................................9
Figure 2.8. Procedure for calculating inverse kinematics .............................................11
Figure 2.9. Solve inverse kinematics problem by geometric method ........................... 12
Figure 2.10. MG996R servo .......................................................................................... 13
Figure 2.11. SG90 servo ................................................................................................ 14
Figure 2.12. ZPT vacuum pad .......................................................................................15
Figure 2.13. Vacuum pump PYP-370 and solenoid valve YQK513 ............................ 15
Figure 2.14. Absorbing workpiece process description ................................................16
Figure 2.15. Releasing workpiece process description .................................................16
Figure 2.16. Kinematic parameters of the robot arm ....................................................17
Figure 2.17. 3D model of the robot arm ........................................................................17
Figure 2.18. Control circuit block diagram ...................................................................18
Figure 2.19. Arduino Nano ............................................................................................ 18
Figure 2.20. LCD 1602 and I2C communication converter module ............................. 20
Figure 2.21. Adapter 5VDC - 1A ..................................................................................21
Figure 2.22. Adapter 24VDC - 2A ................................................................................21
Figure 2.23. LM2596 Module .......................................................................................22
Figure 2.24. Schematic of servo control capacity block ...............................................23
Figure 2.25. Schematic of geared motor capacity block ...............................................23
Figure 2.26. Schematic of vacuum pump and solenoid valve control capacity block ..24
Figure 2.27. Complete control circuit schematic .......................................................... 25
Figure 2.28. PCB design of the control circuit .............................................................. 26
ix
Figure 2.29. 3D model of the control circuit .................................................................27
Figure 2.30. Shape and size of the control box design ..................................................27
Figure 2.31. Webcam Logitech C310............................................................................28
Figure 2.32. 3D model of classification system ............................................................ 28
Figure 3.1. Algorithm of image processing ...................................................................30
Figure 3.2. Algorithm of workpiece detection .............................................................. 31
Figure 3.3. Image in RGB (left) and HSV (right) color space ......................................31
Figure 3.4. Binary image ............................................................................................... 32
Figure 3.5. Binary image after morphological transformation ......................................32
Figure 3.6. Contour and center coordinates of colored region distinguished ...............32
Figure 3.7. Algorithm of robot control ..........................................................................33
Figure 3.8. Algorithm of robot control for lifting and moving the workpiece ..............34
Figure 4.1. Actual system model ...................................................................................35
Figure 4.2. Actual control circuit ..................................................................................35
Figure 4.3. Actual robot arm model ..............................................................................36
Figure 4.4. Absorbing process of robot arm ..................................................................37
Figure 4.5. Feature extraction by detection processing .................................................38
Figure 4.6. LCD screen displays data from the system .................................................40
Figure 4.7. Workpiece data in database.........................................................................40
x
LIST OF ABBREVIATIONS
Abbreviations Explanations
OpenCV Open Source Computer Vision Library
DOF Degrees Of Freedom
I2C Inter-Integrated Circuit
UART Universal Asynchronous Receiver-Transmitter
LCD Liquid Crystal Display
RGB Red, Green, Blue
HSV Hue, Saturation, Value
PID Proportional Integral Derivative
xi
Workpiece classification system using robot arm and computer vision
Chapter 1. PROJECT OVERVIEW
1.1. Introduction
Currently, robotics systems are increasingly developing and playing an important
role in the industry. They include complex and advanced technologies from many fields
such as sensors, mechanics, electronics, automatic control, computer science and
artificial intelligence. Therefore, this is a high-tech industry that influences the future
development of mankind.
Robot arms are used in automated production lines ranging from heavy industries
such as automobile manufacturing to the sorting of consumer products. This is
considered a future labor tool, the main workforce in all fields. Robots greatly assist
humans in production, especially in harsh environments with many toxic and dangerous
substances. Currently, in the world, many new types of robots have appeared, marking
a breakthrough development in this field, helping large enterprises increase sales by
reducing human resources, production costs and increasing productivity.
Therefore, the current trend of education in universities is to promote research
into more advanced technologies in the field of robotics, to approach the robotics
industry of countries that thrive in this industry.
Along with the development of the robotics industry, computer vision is also an
area that is considered to be the trend of the future. Essentially, these systems include
acquiring, processing, and interpreting image-related data for several specific
applications. Computer vision is considered as the eyes of robots, making them easier
and more flexible to operate, helping to handle complex tasks, improving labor
productivity, and reducing dangers to humans.
The combination of computer vision and robotics in the workpiece classifying
systems is being increasingly promoted. These workpiece classification systems can
recognize and classify different objects based on color, shape, size, etc. Using images
from the camera, conducting analysis and processing helps extract useful information
for robots, helping them to execute specific tasks. Thence, we can see that the
programming, control, calculation, image processing and communication between the
computer and the robot are extremely necessary and important.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Phan Chi Vu Supervisor: Dr. Nguyen Le Hoa 1
1.2. Related research

Along with the development of the robotics industry in the world, our country is
also constantly investing in developing and training high-quality human resources to be
able to quickly apply the world's advanced technologies. Therefore, as a leading
technical school in Vietnam, The University of Danang - University of Science and
Technology also constantly promotes the development and expansion of laboratories
and organizes seminars to help students have the opportunity to experience and approach
techniques in this field, apply them into practical application, specifically the workpiece
classifying systems.
Some theses and researches in recent years by students of UD - DUT:
Figure 1.1. Project " Thiết kế và chế tạo cánh tay robot phân loại sản phẩm sử dụng
xử lý ảnh " by UD - DUT students [1]
Figure 1.2. Project "Thiết kế và chế tạo cánh tay robot 3 bậc tự do ứng dụng trong
phân loại sản phẩm bằng xử lý ảnh" by UD - DUT students [2]
Truong Thi Mai Hue
Nguyen Van Anh Quan
1.3. Research objective

The objective of this research is to build a system model that classifies
workpieces that differ only in color on a conveyor using a robot arm and computer
vision. The system consists of 3 functional blocks as shown in the diagram below:
Figure 1.3. Functional block diagram of the system

- The actuator block includes the parts involved in the system's movement. In
which, it is required to design a simple robot arm that performs the function of picking,
moving and releasing the workpiece.
- The image processing block is responsible for acquiring images, researching
suitable image processing algorithms to extract characteristic information of specific
workpieces, and then sending them to control the robot arm.
- The control block is responsible for controlling the actuators in the system, the
most important of which is to control the robot arm based on the information received
from the image processing block to accurately pick up and classify the workpiece
moving on the conveyor through solving the inverse kinematics problem from the
coordinates of the picking position.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Chapter 2. DESIGN A CLASSIFICATION SYSTEM USING ROBOT

ARM AND COMPUTER VISION
2.1. System overview

With the set requirements, a workpiece classification system model is built
including the following parts:
1. Workpiece
2. Workpiece container
3. Conveyor
4. Robot arm
5. Control box
6. Laptop
7. Camera
In which,
- Laptop and camera act as computer vision system that performs image
processing block.
- The control box contains the circuit that controls the actuators, namely the
conveyor and the robot arm.
We proceed to design and select each part of the system.
2.2. Workpiece selection and design
To simulate workpieces with the same size and shape but different in color, we
designed rectangular box-shaped workpieces with 3 colors red, blue and green with the
size as shown in Figure 2.1 below. Each workpiece is made of 3D printing material and
weighs approximately 5g.
Figure 2.1. Shape and size of the workpiece design
Truong Thi Mai Hue

Nguyen Van Anh Quan
After designing the workpiece, we also design a workpiece container matching

the size of the workpiece.
Figure 2.2. Shape and size of the workpiece container

2.3. Conveyor selection
To classify the workpieces designed in Section 2.2 using a robot arm and
computer vision, we select a conveyor with the following parameters:
- Dimension: 60 x 10 cm
- Roller shaft diameter: 4 cm
- Use geared motor 370 L 24VDC with a rotation speed of 57 rpm.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.3. Conveyor with geared motor 370L 24VDC

Table 2.1. Technical specifications of geared motor 370L 24VDC
Parameter Value Unit
Operating voltage 24 VDC
Running current 75 mA
Stall current 1.3 A
Shaft diameter 4 mm
Rotation speed 57 rpm
We can calculate the conveyor’s stable speed using the following formula:
Croller shaft × Sroller shaft
Vconveyor = (cm/s) (2.1)
60
In which,
- Croller shaft : Perimeter of roller shaft [cm]
- Sroller shaft : Rotation speed of roller shaft [RPM]
When the workpiece is placed on the conveyor, because the weight of the
workpiece is extremely small (5g), insignificant compared to the conveyor’s weight, it
can be considered that the conveyor’s speed is the speed of the workpiece.
2.4. Robot arm design
The design of the robot arm should take into account the characteristics of the
system, specifically the operating space, reach and load capacity. The target is to build
a robot arm for the purpose of classifying workpieces on the conveyor along with the
parameters of the workpiece and conveyor which are selected above.
2.4.1. Robot arm structure selection
A workpiece classification system needs fast processing speed, workpieces are
on dynamic conveyors so the robot arm requires flexible handling capabilities, and at
the same time achieves high accuracy.
Truong Thi Mai Hue

Nguyen Van Anh Quan
With the requirements of the research, the robot arm needs an essential quantity
of DOF to accommodate the handling. However, the more DOF, the more complex the
design process as well as the investment cost increase. We need to strike a balance
between these two factors.
Here are some popular robot structures:
a) b)
c) d)
Figure 2.4. Some popular robot arm structures
Thereby, the robot with structure c), a spherical robot is chosen as the foundation
for the design.
For the convenience of the purpose of picking workpieces on the conveyor, we
have chosen an overall structure of the robot arm with 4 rotational joints which ensure
that the last link is always in the direction perpendicular to the conveyor surface.
Figure 2.5. The overall structure of the robot arm
Truong Thi Mai Hue

Nguyen Van Anh Quan
2.4.2. Kinematics problem

After choosing the structure of the robot arm, we need to build a kinematic model
for it.
a) Denavit - Hertenberg representation
To build kinematic equations describing the robot's motion, we model the robot
with the Denavit - Hartenberg representation. In which, each link is considered as a rigid
body connecting two axes of two adjacent joints of a robot arm. The joint axis is defined
by straight lines in space.
- Identify an axis frame for each link.
- Build a general transformation between adjacent coordinate frames
i−1
(homogeneous transformation matrix Ai ).
i−1
- After having the matrices Ai describe the relationship between adjacent links,
we can figure out the relationship between the end-effector and the coordinates
attached to the origin {O}.
Figure 2.6. Modeling links and joints in terms of Denavit - Hartenberg

Denavit – Hartenberg parameters:
- Link parameters: ai , αi ⇒ Perform the transformation across the X i - axis.
1. Kinematic length of link ai is the distance from the Zi−1 - axis to the Zi - axis
measured along the X i - axis.
2. Link twist αi is the angle between the Z i−1 - axis and the Zi measured around
the X i - axis. The positive direction is determined by the right-hand rule.
- Joint parameters: di , θi ⇒ Perform the transformation across the Zi−1 - axis.
1. Kinematic length of joints di là is the distance from the X i−1 - axis to the X i -
axis measured along the Zi−1 - axis.
Truong Thi Mai Hue

Nguyen Van Anh Quan
2. Joint twist θi is the angle between the X i−1 - axis and the X i - axis measured
around the Zi−1 - axis. The positive direction is determined by the right-hand
rule.
To perform the Denavit - Hertenberg representation, we proceed to set up the
coordinate frame on the selected robot arm as follows:
- Select O0 x0 y0 z0 coordinate frame has O0 located at the center of joint axis 1.
- The z0 - axis has the same direction as joint axis 1.
- The z1 - axis has the same direction as joint axis 2.
Since joint axis 1(z0 ) và joint axis 2(z1 ) are perpendicular and intersect so O1 is
located at the intersection of these two joints axis.
- The direction of the x1 - axis is selected by the product of the vector + (z⃗⃗⃗0 × ⃗⃗⃗
z1 ).
- O1 x1 y1 z1 translates along the z0 - axis a distance of d1 relative O0 x0 y0 z0 .
- Select the x0 - axis in the same direction as the x1 - axis.
- The y0 - axis and the y1 - axis are chosen according to the right-hand rule.
- O2 x2 y2 z2 translates along the x1 - axis a distance of a2 .
- O3 x3 y3 z3 translates along the x2 - axis a distance of a3 .
- O4 x4 y4 z4 is placed at the end point of the end-effector and is a distance of a4
away from the center of joint axis 4.
Figure 2.7. Identify the coordinate system for the robot arm
From the built coordinate system, we get the Denavit - Hertenberg table as below.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Table 2.2. Denavit - Hertenberg parameters

Link θi αi ai di
π
1 θ1 0 d1
2
2 θ2 0 a2 0
3 θ3 0 a3 0
π
4 − − (θ2 + θ3 ) 0 a4 0
2
b) Forward kinematics
Forward kinematics is the problem of finding the end-effector position and
orientation from the specified values of joints (𝜃1 , 𝜃2 , 𝜃3 , 𝜃4 , 𝜃5 , 𝜃6 … ) → (x, y, z, α,
β, γ…).
After representing Denavit - Hartenberg, we have the transformation matrices:
i−1
Ai = Rot z (θ)Transz (d)Transx (a)Rot x (α)
cosθi −sinθi 0 0 1 0 0 0 1 0 0 ai 1 0 0 0
sinθi cosθi 0 0 0 1 0 0 0 1 0 0 0 cosαi −sinαi 0
= [ ][ ][ ][ ]
0 0 1 0 0 0 1 di 0 0 1 0 0 sinαi cosαi 0
0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1
cosθ −sinθ cosα sinθ sinα a cosθ (2.2)
sinθ cosθ cosα −cosθ sinα a sinθ
= [ ]
0 sinα cosα d
0 0 0 1
From this we can calculate the homogeneous transformation matrix for the robot
arm:
0
Tn = 0 A1 1 A2 3 A3 … n−1
An (2.3)
Thus, after obtaining the Denavit - Hartenberg paramete defined in Table 2.2.
above, we can calculate the transformation matrices for each link as follows:
Notation: cos(θi ) = ci , sin(θi ) = si , cos(θi + θj ) = cij , sin(θi + θj ) = sij
c1 0 s1 0 c2 −s2 0 a 2 c2
0 s 0 −c1 0 1 s c2 0 a2 s2
A1 = [ 1 ] A2 = [ 2 ]
0 1 0 d1 0 0 1 0
0 0 0 1 0 0 0 1
c3 −s3 0 a3 c3 −s23 c23 0 −a4 s23
2 s c3 0 a3 s3 3 −c −s23 0 −a4 c23
A3 = [ 3 ] A4 = [ 23 ]
0 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1
Truong Thi Mai Hue

Nguyen Van Anh Quan
c1 c2 −c1 s2 s1 a2 c2 c1
s c −s1 s2 −c1 a2 c2 s1
⇒ 0 A2 = 0 A1 1 A2 = [ 1 2 ]
s2 c2 0 a2 s2 + d1
0 0 0 1
c1 c23 −c1 s23 s1 c1 (a2 c2 + a3 c23 )
s c 3 −s1 s23 −c1 s1 (a2 c2 + a3 c23 )
⇒ 0 A3 = 0 A1 1 A2 2 A3 = [ 1 2 ]
s23 c23 0 d1 + a2 s2 + a3 s23
0 0 0 1
0 c1 s1 c1 (a2 c2 + a3 c23 )
0 s1 −c1 s1 (a2 c2 + a3 c23 )
⇒ 0 T4 = 0 A3 3 A4 = [ ]
−1 0 0 d1 − a4 + a2 s2 + a3 s23
0 0 0 1
0
From the homogeneous transformation matrix T4, we can know where the end-
effector point is:
O4 (c1 (a2 c2 + a3 c23 ), s1 (a2 c2 + a3 c23 ), d1 − a4 + a2 s2 + a3 s23 )
c) Inverse kinematics
Inverse kinematics is the problem of determining the joint angles when knowing
the position of the end-effector (x, y, z, α, β, γ…) → (𝜃1 , 𝜃2 , 𝜃3 , 𝜃4 , 𝜃5 , 𝜃6 … ).
The complexity of the inverse kinematics problem increases as the number of
DOF increases.
To solve the inverse kinematics problem, two main methods are used, that is the
geometric method (if the robot has a special configuration) or the algebraic method
(using transformations to give the equation of joint variables).
Here is the procedure for calculating inverse kinematics for the robot arm:
Figure 2.8. Procedure for calculating inverse kinematics

Because the selected robot model has a simple structure, we have chosen a
geometric method to solve the inverse kinematics problem.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Assume the point the robot arm wants to reach has coordinates (𝑃𝑥 , 𝑃𝑦 , 𝑃𝑧 )
Figure 2.9. Solve inverse kinematics problem by geometric method

- Based on the image above, it can be seen that:
Py
θ1 = arctan ( )
Px
- Applying the cosine formula in triangle ABC, we have:

AB2 + BC2 − AC2 AB2 + BC2 − (AH2 + HC2 )
cos(π − θ3 ) = =
2. AB. BC 2. AB. BC
2 2 2 2
a2 + a3 − (Px + Py + (Pz − d1 + a4 )2
=
2a2 a3
Px 2 + Py 2 + (Pz − d1 + a4 )2 − a22 − a23
⇒ cos(θ3 ) =
2a2 a3
Px 2 + Py 2 + (Pz − d1 + a4 )2 − a22 − a23
⇒ θ3 = ∓ arccos ( )
2a2 a3
Based on Figure 4.8., θ3 < 0, we choose a negative value:
Px 2 + Py 2 + (Pz − d1 + a4 )2 − a22 − a23
θ3 = −arccos ( )
2a2 a3
- Calculate θ2 :
HC KC
̂ = HAC
θ2 = HAK ̂ + CAK
̂ = arcsin ( ) + arctan ( )
AC AB + BK
Pz − d1 + a4 a3 . sin(|θ3 |)
= arcsin + arctan ( )
a2 + a3 cos(θ3 )
√Px 2 + Py 2 + (Pz − d1 + a4 )2
( )
Truong Thi Mai Hue

Nguyen Van Anh Quan
2.4.3. Transmission system selection for rotational joints

Electromechanical transmission is selected because it allows ensuring high
displacement accuracy and the ability to perform complex movements. Specifically, the
electrical device selected to provide movements for the robot is the MG996R RC servo.
Figure 2.10. MG996R servo

Table 2.3. Technical specifications of MG996R servo
Operating voltage 4.8 - 7.2 V
Running current 500 - 900 mA
No-load speed 0.17 s/60°
Peak stall torque 9.4 kg-cm
MG996R servo has a stall torque of 9.4 kg-cm, which means the servo can
withstand a load of 9.4kg at a distance of 1 cm from the center of the servo’s shaft.
When the distance is doubled, the weight of the load is halved.
Since the weight of the workpiece is quite small (approximately 5g), the load
acting on the servo is mainly the links of the robot arm. This is the main factor in
choosing the material of the links to ensure the load capacity of the selected servo motor.
2.4.4. End-effector selection
The end-effector, known as the gripper, is the part that directly performs the work
of the robot arm.
The requirements of the gripper of the robot arm are not too demanding on the
movement mechanism, but it is necessary to ensure some factors such as gripping the
right workpiece, holding it firmly but not damaging the workpiece, in addition, it also
needs to be compact and quickly impact.
With the rectangular-box shape of the workpiece, we chose a vacuum gripper
with a rotational wrist to orient the workpiece.
Truong Thi Mai Hue

Nguyen Van Anh Quan
SG90 servo is selected to create a circular motion for the gripping wrist.
Figure 2.11. SG90 servo

Table 2.4. Technical specifications of SG90 servo
Operating voltage 4.8 - 6 V
Running current 500 - 900 mA
No-load speed 0.12 s/60°
Peak stall torque 1.5 kg kg-cm
Because the wrist part only bears the load from the gripper and the workpiece,
there is no need for large stall torque like other rotational joints of the robot.
The gripper is made up of a vacuum pad combined with a solenoid valve and a
vacuum pump. The lifting force of the vacuum gripper can be found by equation:
F=P×A (2.4)
In which,
- F: The weight of the objects [kg] multiplied by the safety factor (A safety factor
of 2 is recommended when the pad face is in a horizontal position)
- P: The vacuum pressure [bar]
- A: The area of the vacuum pad measured [cm2]
With a circular vacuum pad:
2 mworkpiece = P × πr 2 , mworkpiece = 5g
This is the basis to choose the following components:
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.12. ZPT vacuum pad
Figure 2.13. Vacuum pump PYP-370 and solenoid valve YQK513

Table 2.5. Technical specifications of ZPT vacuum pad

Diameter of pad 1 cm
Table 2.6. Technical specifications of vacuum pump PYP-370

Operating voltage 3.7 - 5 V
Current consumption 400 mA
Pressure range 60 - 100 kPa
Airflow 3.2 L/m
Table 2.7. Technical specifications of solenoid valve YQK513

Operating voltage 24 V
Current consumption 130 mA mA
Capacity < 3.5 W
Pressure range 0 - 250 mmHg
Theory of operation of the gripper:
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.14. Absorbing workpiece process description
Figure 2.15. Releasing workpiece process description

When the solenoid valve is not supplied, ports 1 and 3 are connected while port 2
is closed. When the vacuum pump is on, the air is absorbed from the vacuum pad through
port 3, to port 1 and the pump’s inlet, then out to the outlet so that the robot arm can lift
the workpiece.
When the pump is off and the solenoid valve is on, port 2 and port 3 are connected
while port 1 is closed. Air will go from port 2 to port 3, through the vacuum pad, and go
out so that the robot arm can drop the object.
2.4.5. Material and dimension selection
Based on the above calculations and selections, mica acrylic sheets with a
thickness of 5 mm are selected to be the material to build the robot arm. This material
is light in weight, low cost, and suitable for the size and weight of the designed
workpiece.
Below are pictures of the robot arm that we designed, the corresponding
kinematic parameters are shown in Figure 2.16.:
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.16. Kinematic parameters of the robot arm
Figure 2.17. 3D model of the robot arm

Details of each link size are presented in the robot arm design drawing in the
Appendix.
Truong Thi Mai Hue

Nguyen Van Anh Quan
2.5. Control circuit design

Figure 2.18. below describes the control circuit block diagram.
Figure 2.18. Control circuit block diagram

2.5.1. Central processing block
In this thesis, the central microcontroller Arduino Nano is selected.
Figure 2.19. Arduino Nano

Arduino Nano is designed on the ATMega328 SMD microcontroller platform
with a small size, but it is fully integrated with the required features of a microcontroller
board such as:
- Oscillating circuit
- Reset circuit
- Power circuit
- Uploading circuit and TTL communication with the computer.
- LED signaling system
Truong Thi Mai Hue

Nguyen Van Anh Quan
Table 2.8. Technical specifications of Arduino Nano

Micro Controller ATmega328
Architecture AVR
Operating Voltage 5V
Flash Memory 32 KB of which 2 KB used by the bootloader
SRAM 2 KB
Clock Speed 16 MHz
Analog Input Pins 8
EEPROM 1 KB
DC Current Per I/O Pins 40 mA
Input Voltage 7 - 12 V
Digital I/O Pins 22
PWM Output 6
Power Consumption 19 mA
Based on the number of devices that need to be controlled and the basic
parameters of the Arduino Nano, it is a perfect choice.
2.5.2. Communication block
a) UART communication
Arduino and laptop communicate with each other via the UART protocol.
Arduino Nano has the built-in IC CH340 to convert USB - UART, so to transmit and
receive data between these 2 devices, we can use a USB cable.
b) I2C communication
Workpiece classification results will be displayed on LCD 1602. This LCD is
capable of displaying several types of characters (text, number, or graphic characters),
easy to put into an application circuit, consumes very little system resources, and has a
low cost.
To connect easily and avoid occupying many pins of the Arduino, we have
chosen an I2C communication converter module that supports LCD connected to the
microcontroller via I2C communication.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.20. LCD 1602 and I2C communication converter module

Table 2.9. Technical specifications of LCD 1602
Operating Voltage 5 VDC
Dimensions 80 x 36 x 12.5 mm
Signal wires 6 pin
Table 2.10. Technical specifications of I2C communication converter module

Operating Voltage 2.5 - 6 VDC
Supported display LCD1602, 1604, 2004 -
Communication I2C -
2.5.3. Power supply block

Besides choosing a power supply for the Arduino, we need to design a separate
power supply to provide for the servo, conveyor, vacuum pump, and solenoid valve to
operate because these components consume quite large currents, need to be powered
isolatedly from theArduino, avoid causing interference when operating.
According to the rated parameters, the solenoid valve and the geared motor
controlling the conveyor operate at 24VDC, so we choose a power supply that is capable
of providing 24VDC and has a high current limitation.
For the servo and vacuum pump, these devices operate at 5VDC, so we selected
a bucking circuit module to step down from the 24VDC selected above to a stable
5VDC.
a) Power supply for Arduino
The adapter has the role of converting AC voltage into DC voltage for DC
equipment.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Adapter 5VDC - 1A is used quite a lot for multiple devices such as network
modems, converters, or power supplies for microcontrollers. With a compact design
along with a high-quality current drop protection circuit, it is suitable for powering
devices and extending their lifespan.
Figure 2.21. Adapter 5VDC - 1A

Table 2.11. Technical specifications of Adapter 5VDC - 1A
Input voltage 100 - 240 VAC
Output voltage 5 VDC
Output maximum current 1 A
b) Power supply for geared motor and solenoid valve

Adapter 24VDC - 2A is a type of switching mode power supply, which can
convert from 220VAC to 24VDC, commonly used to power control circuits and control
modules using an external supply. With the current limit up to 2A, Adapter 24VDC -
2A can provide voltage for the solenoid valve and the geared motor controlling the
conveyor to work well.
Figure 2.22. Adapter 24VDC - 2A
Truong Thi Mai Hue

Nguyen Van Anh Quan
Table 2.12. Technical specifications of Adapter 24VDC - 2A

Input voltage 100 - 240 VAC
Output voltage 24 VDC
c) Power supply for servo and vacuum pump

All of the servos and the vacuum pump operate at 5VDC while we already have
the 24VDC supply selected above, which requires a buck converter that is responsible
for stepping down voltage from 24VDC to 5VDC and satisfies these technical
specifications:
- Input voltage: 24VDC
- Output voltage: 5VDC
- Output current: > 1.5A
With such requirements, the LM2596 module is a perfect choice.
Figure 2.23. LM2596 Module

Table 2.13. Technical specifications of LM2596 module
Input voltage 4 - 35 VDC
Output voltage 1 - 30 VDC
Truong Thi Mai Hue

Nguyen Van Anh Quan
2.5.4. Capacity block

a) Servo control capacity block
Figure 2.24. Schematic of servo control capacity block

- Phototransistor PC817 (U4) is inserted to isolate the 5VDC supply for Arduino
and the 5VDC supply of the LM2596, avoiding the phenomenon that when the
servo or the vacuum pump is operating, the generated current is quite large,
causing interference with the Arduino.
- When U4 is not conductive, resistor R5 is responsible for pulling the servo
control signal down to 0.
- When there is a control signal from the Arduino, resistor R6 has the function to
limit the current through the LED inside U4, making this LED operate. At that
time, the light detector makes the BJT inside U4 conduct.
- Thereby, the signal pinout of the servo is supplied 5VDC.
b) Geared motor control capacity block
Figure 2.25. Schematic of geared motor capacity block

- TA6586 H bridge IC has the function to control the geared motor via 2 pinouts
INA and INB.
Truong Thi Mai Hue

Nguyen Van Anh Quan
- When the signal of these two pins is opposite, the motor runs in a certain direction
(for example, when INA level 1 and INB level 0, run in the forward direction and
vice versa).
- In this project, we only need the conveyor to rotate in a certain direction, so we
fixed the INB pin to always be at 0, then put the PWM control pin on the Arduino
into the INA to control the geared motor with the pulse width modulation method,
thereby we can control the speed of the conveyor.
- Similar to the servo control power block, when there is a control signal, the INA
pin will be pulsed to control the desired motor speed.
c) Vacuum pump and solenoid valve control capacity block
Figure 2.26. Schematic of vacuum pump and solenoid valve control capacity block
- Phototransistors PC817 (U4, U11) are inserted to isolate the 5VDC supply for
Arduino and the 5VDC, 24VDC supply of the LM2596, avoiding the
phenomenon that when the pump or the valve is operating, the generated current
is quite large, causing interference with the Arduino.
Truong Thi Mai Hue
Nguyen Van Anh Quan
- When there is a control signal from the Arduino, resistors R5 and R17 have the
function to limit the current through the LED inside U4 and U11, making these
LEDs operate. At that time, the light detector makes the BJT inside U4 and U11
conduct. Thereby, resistors R6 and R18 limit the current through Q3 and Q4 BJT
and conduct the RL2 and RL4 relays, so the pump and the valve can operate.
- LED D5 and D7 help notify if there is a control signal.
- Diodes D6 and D8 consume the electromagnetic induction current generated by
the coil in the relay when the relay loses power, which help to protect the BJT
from this current.
2.5.5. Complete control circuit schematic
Figure 2.27. Complete control circuit schematic

With Arduino Nano as the central microcontroller for the circuit, the control
pinouts are used to perform movements of the robot arm as well as turn on and off the
components in the system and display the necessary parameters on the LCD screen.
Truong Thi Mai Hue

Nguyen Van Anh Quan
- 5 and 6 digital pinouts have the function to control the vacuum pump and the
solenoid valve.
- 7, 8, 9 and 10 digital pinouts are responsible for controlling the servo attached to
the robot arm.
- 11 PWM pinout is used to control the speed of the geared motor by the pulse
width modulation method.
- TX and RX pinouts help the Arduino to communicate with the laptop by UART
communication.
- LCD is controlled through I2C communication converter module, connecting
SDA, SCL pins to A4, A5 analog pinouts.
- The power supply block consists of two separate blocks:
+ 24VDC supply from Adapter 24VDC - 2A which powers the geared motor
controlling the conveyor, the solenoid valve and 5VDC from LM2596 module supplies
for servos and the vacuum pump.
+ 5VDC supplies for Arduino from Adapter 5VDC - 1A.
Below are the pictures of the PCB design and 3D model of the control circuit.
Figure 2.28. PCB design of the control circuit
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 2.29. 3D model of the control circuit

A control box is designed to contain the control circuit.
Figure 2.30. Shape and size of the control box design

Truong Thi Mai Hue
Nguyen Van Anh Quan
2.6. Camera selection

The camera plays an important role as an image acquisition device. In this thesis,
we use the Logitech C310 Webcam, a camera model developed by Logitech, suitable
for use in simple computer vision models thanks to its low cost and appropriate
configuration.
Figure 2.31. Webcam Logitech C310

Table 2.14. Technical specifications of webcam Logitech C310
Resolution 720p/30fps
Camera pixels 1.2 million pixels
Connector USB - A
Focal length Permanent
Operating system support Windows, MAC - OS,...
2.7. Complete model of the system

After selecting and designing all the parts, we get a model of the entire
classification system as shown below.
Figure 2.32. 3D model of classification system
Truong Thi Mai Hue

Nguyen Van Anh Quan
Chapter 3. CONTROL ALGORITHM
3.1. The main idea

The workpiece classification system using a robot arm and computer vision
focuses on studying the control of the robot arm based on the data returned from the
computer vision system.
Specifically, the computer vision part is responsible for determining the color and
coordinates of the workpiece in pixels.
After that, we need to convert the determined coordinates to the robot coordinates
frame to execute the picking. Therefore, the camera system has to be capable of
detecting the workpiece on the frame and determining the coordinates of the center of
the workpiece in the preselected coordinate frame.
To solve this workpiece detection task, the static image processing technique is
used, ie process each frame received from the camera, combining the coordinate
information of the workpiece at the time the camera detects it, along with the speed of
the conveyor and the motion time of the robot arm, we can calculate the position where
the robot arm can pick up the workpiece.
In addition, data on classified workpieces will also be stored in the database for
later management purposes. The selected database is MySQL.
To build this computer vision system, we use the OpenCV library and Python
language to program the processing program and communicate with the control block
of the system, namely Arduino.
3.2. Image processing algorithm
Below is the image processing algorithm used in the thesis.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 3.1. Algorithm of image processing

When the program starts, the camera is turned on and the laptop proceeds to read
each frame the camera sends. When receiving the ‘Start detection process’ command,
the laptop starts the detection process.
If there is a workpiece with the color that matches the selected color in the
readable frame, the program will extract the feature data about coordinates, tilt angle
and color of the workpiece. The obtained workpiece coordinates are converted to the
robot coordinate system and then send the ‘Execution’ command to the Arduino. After
that, the program stops the detection process. This progress ensures that data is only sent
to the Arduino once when a workpiece is detected.
Truong Thi Mai Hue

Nguyen Van Anh Quan
If no workpiece is detected, the laptop sends a command to control the robot arm
to its original position, called the ‘Home’ command.
Figure 3.2. Algorithm of workpiece detection

Figure 3.2. above demonstrates the flow of workpiece detection algorithm.
In this process, the technique to be used is contouring. Contouring is a computer
vision technique that simply joins all continuous points of the same color and intensity.
The workpiece detection process is performed step by step as follows:
- Define the range of each selected color to classify in HSV color space. This range
depends on the color of the designed workpiece.
- Convert each image frame is readable by the camera in RGB color space to HSV
color space because the HSV space allows for more precise color selection.
Figure 3.3. Image in RGB (left) and HSV (right) color space
Truong Thi Mai Hue

Nguyen Van Anh Quan
- Create a corresponding mask for each range selected color. After this step, the
image is binarized, the desired color is assigned a value of 1, and the remainder
of the image is assigned a value of 0.
Figure 3.4. Binary image

- Perform morphological transformation in this image to remove noises.
Figure 3.5. Binary image after morphological transformation

- Draw contour to display the colored region distinguished.
- Find the coordinates and tilt angle of created contour. This is the coordinates and
tilt angle of the workpiece.
Figure 3.6. Contour and center coordinates of colored region distinguished
Truong Thi Mai Hue

Nguyen Van Anh Quan
3.3. Robot arm control algorithm

The process of controlling the robot is shown in Figure 3.7..
Figure 3.7. Algorithm of robot control

When the program starts, check button status. If this button is pressed, the
Arduino will send ‘Start detection process’ command to laptop and then wait to receive
the response command from the laptop.
If no workpiece is detected, Arduino will perform the ‘Home’ command.
Opposite, if a workpiece has the same color as the selected color is detected, the
Arduino will perform the process of lifting and moving the workpiece according to the
‘Execution’ command with the received data of coordinates, tilt angle and color. This
process is described in the Figure 3.8. below.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 3.8. Algorithm of robot control for lifting and moving the workpiece
From the received coordinates of the workpiece, Arduino calculates the value of
joint variables of the robot arm after solving the inverse kinematics problem in Section
2.4.2 above.
Next, we calculate the time for the robot arm to move from the current position
to the position where the workpiece is detected (t robot_motion ). This time is equal to the
angle at which the robot arm rotates divided by the speed of each joint.
So if the robot arm moves to the position where the workpiece is deteced, the
workpiece has moved to a new position ynew = t robot_motion × vworkpiece as the
conveyor only moves in a certain direction (y-axis), the x-axis and z-axis coordinates of
the workpiece are unchanged.
Arduino controls the robot arm to the new position of the workpiece and then
proceeds to lift it at the time t = t robot_motion , then control the gripper to rotate the wrist
based on the tilt angle of the workpiece and move it to the corresponding workpiece
container.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Chapter 4. EXPERIMENTAL RESULTS AND EVALUATION
After designing and building the algorithm for the workpiece classification
system model, we conduct experiments to test the system and evaluate the obtained
results.
4.1. Experimental results
The actual system model that we have built is shown in Figure 4.1. below.
Figure 4.1. Actual system model

4.1.1. Control circuit
Below is the actual image of the control circuit:
Figure 4.2. Actual control circuit

Truong Thi Mai Hue
Nguyen Van Anh Quan
4.1.2. Robot arm
Figure 4.3. Actual robot arm model

a) Joint movement
We carried out checking the robot arm’s joint range of movement.
Table 4.1. Joints movable range of robot arm
Parameter Value
Joint 1 [-100°, 100°]
Joint 2 [5°, 95°]
Joint 3 [-155°, -75°]
b) Impact of vacuum gripper
Check the operation of the vacuum gripper of the robot arm during absorbing and
releasing process.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 4.4. Absorbing process of robot arm

c) Robot arm control
After solving the inverse kinematics problem, the robot arm is controlled to move
to any random position when knowing the coordinates of that point. The desired position
of the destination point and the actual position are measured by a ruler, the results are
shown in the following table.
Table 4.2. Experimental results of robot arm control
Desired position of Actual position of
Error
gripper gripper
Times (cm)
measurements (cm) (cm)
𝑃𝑥 𝑃𝑦 𝑃𝑧 𝑃𝑥 𝑃𝑦 𝑃𝑧 |∆𝑒𝑃𝑥 | |∆𝑒𝑃𝑦 | |∆𝑒𝑃𝑧 |
1 18 15 5 17.6 14.6 5.6 0.4 0.4 0.6

2 20 10 2 21.1 10.5 1.8 1.1 0.5 0.2
3 25 0 0 26.2 1 0.3 1.2 1 0.3
4 18 18 5 18.7 18.2 4.9 0.7 0.2 0.1
5 13 12 10 14 12.8 9.4 1 0.8 0.6
From Table 4.2, the average error for each axis is:
Truong Thi Mai Hue

Nguyen Van Anh Quan
̅̅̅̅̅̅̅̅ ∑5i=1|∆ePx |
|∆ePx | = = 0.88 (cm)
5
̅̅̅̅̅̅̅̅ ∑5i=1 |∆ePy |
|∆ePy | = = 0.58 (cm)
5
̅̅̅̅̅̅̅̅ ∑5i=1|∆ePz |
|∆ePz | = = 0.36 (cm)
5
4.1.3. Workpiece detection algorithm
The workpiece is fixed on the conveyor, then use a ruler to measure the initial
position of the center of the workpiece in the x and y axes, then start running the program
to identify the color, the coordinates of the center, and the tilt angle of the workpiece
and convert the detection coordinates to the robot coordinates to check the implemented
algorithm and the accuracy of the received data.
Figure 4.5. Feature extraction by detection processing

To be objective, the system will be tested in different lighting conditions using a
light sensor to measure light intensity.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Table 4.3. Experimental results of workpiece detection algorithm in different lighting

conditions
Times measurements 1 2 3 4 5 6
Light intensity (LUX) 150 220
Color Blue Red Green Blue Red Green
Actual 𝑃𝑥 20.3 22.2 19.4 21.3 25.8 20
Center
feature
coordinates (cm) 𝑃𝑦 17.5 15.9 13.5 15.6 18.4 19.5
value
Orientation angle (°) 44 90 120 70 30 75
Feature Color Blue Red Green Blue Red Green
value 𝑃𝑥 20.21 21.34 19.23 21.25 25.67 20.26
Center
extracted by
coordinates (cm) 𝑃 16.6 15.44 13.45 15.81 18.67 19.56
detection 𝑦
processing Orientation angle (°) 45 89 122 69 30 75

Color Right Right Right Right Right Right
Center 𝑃𝑥 0.09 0.86 0.17 0.05 0.13 0.26
Detection coordinates error
results (cm) 𝑃𝑦 0.9 0.46 0.05 0.21 0.27 0.06
Orientation angle
1 1 2 1 0 0
error (°)
4.1.4. Classification system model

Conduct an operational check of the entire workpiece classifying system with the
conveyor operating at 4 cm/s.
The LCD screen displays the workpiece identification data which is sent from
the laptop to the Arduino every time a workpiece is detected by the camera and displays
the number of workpieces placed in the workpiece container by color.
Truong Thi Mai Hue

Nguyen Van Anh Quan
Figure 4.6. LCD screen displays data from the system

Randomly put the workpiece on the conveyor, the average number of workpieces
classified by the system model in a minute is 6 workpieces.
The accuracy test results are shown in the following table:
Table 4.4. Experimental results of workpiece classification system in different lighting
conditions
Light Number of Correct
Number of
intensity correct classification Note
workpieces
(LUX) classification rate
180 30 26 87% The robot arm missed 4 times
220 30 28 94% The robot arm missed 2 times
The robot arm missed 4 times
250 30 25 84%
1 wrong color detection
The data of the workpiece which the system has successfully classified is saved
to the database.
Figure 4.7. Workpiece data in database

Truong Thi Mai Hue
Nguyen Van Anh Quan
4.2. Experimental evaluation

4.2.1. Operation of control circuit
The control circuit has been fully designed and works well, there is no
interference thanks to the power supply isolation.
4.2.2. Operation of robot arm
a) Movement range and end-effector impact
The robot arm has a sufficient movement range, the vacuum gripper performs
absorbing, holding, and releasing workpieces well with the designed workpiece.
b) Robot arm control
Because the mechanical design is not completely accurate and optimized, a few
uncertain joints along with mica acrylic plastic material have led to errors as calculated
above.
Average error of the position control in x, y, z axes is less than 9 mm.
4.2.3. Workpiece detection algorithm
The test results show that the algorithm works well in different lighting
conditions:
- The experimental color identification rate of the non-moving workpiece is 100%.
- The results of detecting the tilt angle and center coordinates of the workpiece in
the x and y axes are also very accurate, the error obtained is very small.
4.2.4. Operation of classification system model
When classifying moving workpieces on a conveyor running at 4 cm/s, color
detection is quite exact in different lighting conditions, but sometimes the robot arm still
misses the workpiece.
Factors that affect the results:
- Due to the error in the process of finding the coordinates of the workpiece and
the rounding process during the calculation, may lead to the value of the joint
variables of the robot arm with accumulated errors.
- The problem of the mechanical structure of the designed robot arm is mentioned
in Section 4.2.2. above.
- Conveyor is not controlled to stabilize the speed, so the actual speed of the
workpiece moving on the conveyor is not exactly as set.
Truong Thi Mai Hue

Nguyen Van Anh Quan
CONCLUSION
1. Achieved results
- Successfully build a model of the workpiece classification system by color using
a robot arm and computer vision.
- Successfully design control circuit for stable operation.
- Successfully design and built a 4-DOF robot arm with vacuum gripper.
- Control the robot arm to a given position according to inverse kinematics with an
error of less than 9 mm.
- Apply image processing techniques to detect and extract features of workpieces.
- Display system data on the LCD screen.
- Store the workpiece data after classifying into the database.
- The model's workpiece classification rate is quite high, approximately 84% in
different lighting conditions.
2. Limitation
- The mechanical design of the robot arm is not completed.
- The robot arm's dynamic model has not been considered yet.
- The movement time of the robot arm is long because the servos are controlled
sequentially, which leads to the system classification speed is not high.
- The classification is only limited to the color criteria of the workpiece with 3
certain colors: Red, Blue, and Green.
- The measurement of the robot's parameters is carried out with a ruler, so the
accuracy is not high.
- There is no user interface for the system model.
3. Orientations of development
- Consolidate the structure of the robot arm by replacing the mica acrylic plastic
material with a metal material.
- Build the dynamic model to increase the accuracy in designing and controlling
the robot arm.
- Build an algorithm to control servos at the same time to increase the classification
efficiency of the system.
- Stable the conveyor’s speed by PID and Fuzzy PID controller.
Truong Thi Mai Hue
Nguyen Van Anh Quan
- Build a classification algorithm based on many other properties of the workpiece.

- Build and develop an intuitive control display.
Truong Thi Mai Hue

Nguyen Van Anh Quan
REFERENCES
[1] Lương Thế Phi, Huỳnh Minh Lựu, Thiết kế và chế tạo cánh tay robot phân loại sản
phẩm sử dụng xử lý ảnh. Da Nang: The University of Danang - University of Science
and Technology, 2019.
[2] Đặng Hữu Tuyến, Bùi Đăng Quân, Thiết kế và chế tạo cánh tay robot 3 bậc tự do
ứng dụng trong phân loại sản phẩm bằng xử lý ảnh. Da Nang: The University of Danang
- University of Science and Technology, 2019.
[3] John J. Craig, Introduction to Robotics - Mechanics and Control, 3rd ed. London:
Pearson Education International, 2005.
[4] Nguyễn Trường Thịnh, Giáo trình kỹ thuật Robot. Ho Chi Minh: Hochiminh
National University Publishing House, 2014.
[5] Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani, Giuseppe Oriolo, Robotics:
Modelling, Planning and Control. New York: Spinger, 2008.
Truong Thi Mai Hue

Nguyen Van Anh Quan
APPENDIX
Drawing of robot arm detailed design

Workpiece Classification Using Robot Arm and Computer Vision

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Workpiece Classification Using Robot Arm and Computer Vision

Uploaded by

Copyright:

Available Formats

THE UNIVERSITY OF DA NANG

UNIVERSITY OF SCIENCE AND TECHNOLOGY

WORKPIECE CLASSIFICATION SYSTEM

STUDENTS: TRUONG THI MAI HUE, 122170072

Supervisor: Dr. NGUYEN LE HOA

GRADUATION PROJECT REQUIREMENTS

Index Student Student No. Class Major

1. Name of the project:

Da Nang, July 5th, 2022

Head of Division Supervisor

We are currently practicing and working on a graduation project at Oanh Vu

Table 2.1. Technical specifications of geared motor 370L 24VDC................................ 6

Chapter 1. PROJECT OVERVIEW

Truong Thi Mai Hue

1.2. Related research

1.3. Research objective

Figure 1.3. Functional block diagram of the system

Truong Thi Mai Hue

Chapter 2. DESIGN A CLASSIFICATION SYSTEM USING ROBOT

2.1. System overview

Figure 2.1. Shape and size of the workpiece design

Truong Thi Mai Hue

After designing the workpiece, we also design a workpiece container matching

Figure 2.2. Shape and size of the workpiece container

Truong Thi Mai Hue

Figure 2.3. Conveyor with geared motor 370L 24VDC

Truong Thi Mai Hue

Figure 2.5. The overall structure of the robot arm

Truong Thi Mai Hue

2.4.2. Kinematics problem

Figure 2.6. Modeling links and joints in terms of Denavit - Hartenberg

Truong Thi Mai Hue

Truong Thi Mai Hue

Table 2.2. Denavit - Hertenberg parameters

Truong Thi Mai Hue

Figure 2.8. Procedure for calculating inverse kinematics

Truong Thi Mai Hue

Figure 2.9. Solve inverse kinematics problem by geometric method

- Applying the cosine formula in triangle ABC, we have:

Truong Thi Mai Hue

2.4.3. Transmission system selection for rotational joints

Figure 2.10. MG996R servo

Truong Thi Mai Hue

Figure 2.11. SG90 servo

Truong Thi Mai Hue

Figure 2.12. ZPT vacuum pad

Figure 2.13. Vacuum pump PYP-370 and solenoid valve YQK513

Parameter Value Unit

Table 2.6. Technical specifications of vacuum pump PYP-370

Table 2.7. Technical specifications of solenoid valve YQK513

Theory of operation of the gripper:

Truong Thi Mai Hue

Figure 2.14. Absorbing workpiece process description

Figure 2.15. Releasing workpiece process description

Truong Thi Mai Hue

Figure 2.16. Kinematic parameters of the robot arm

Figure 2.17. 3D model of the robot arm

Truong Thi Mai Hue

2.5. Control circuit design

Figure 2.18. Control circuit block diagram

Figure 2.19. Arduino Nano

Truong Thi Mai Hue

Table 2.8. Technical specifications of Arduino Nano